Archive for the ‘Tech’ Category

« Older Entries

Home Theater Without Breaking the Bank – Part 2

Wednesday, January 13th, 2010

One Remote to Rule Them All

This is part two of a multi-part story on upgrading my home stereo. Part 1 was about finding the right solution for video playback.

After getting video sorted out, the next step was to find some way to reduce the jumble of remote controls I was currently using. At this point, I had all sorts of crap to remotely control the devices: a wireless keyboard for the HTPC; remotes for the cable box / DVR, the receiver, and the TV, Playstation 3 em Playstation 2 remotes, and Wii remotes. Surely, I could do something to reduce this remote madness?

My first idea was to use the remote that came with my receiver. It could control up to nine devices, and from reading the manual I gathered that it could control many different types. The litmus test would be whether or not my receiver’s remote could control my cable box / DVR. It’s a Pace box, which is one brand I’d never heard of before. I was quite pleased to see the company name even listed in my remote’s manual. It was worth a shot.

One neat thing my receiver does (it’s an Onkyo HT-RC160) is on-screen configuration of remote codes for controlling other devices. That means it displays instructions and the codes themselves right on the TV. It sure beats the heck out of flipping the remote over (to read the instructions) for the first few codes, or (even worse) digging out the manual and looking up the codes. This made it easier to try the codes — there were about ten of them — but it turned out to be for nought, because none of the codes worked. Back to the drawing board.

A couple of years ago I got my dad a Logitech Harmony 659 remote for Christmas. The Harmony series allows you to define ‘activities’ that are essentially macros for the devices it controls. You can define one called “Watch a Movie” that turns on your receiver, TV, and DVD player, and switches everything to the right inputs. This was a great gift because it was something he could tinker with and it would make things a lot easier for my mom, who doesn’t want to deal with all the devices, inputs, and buttons. I thought this might be the perfect solution for my problem.

So I went to Target and made the most impulsive purchase of my home theater-upgrading spree. Luckily, they had one Harmony 620, which is precisely the remote I needed. I could have ordered it online (that’s how I got most of the equipment I needed for this upgrade), but didn’t want to have to wait. It was a Saturday, which meant that the earliest I would have gotten the thing would have been Wednesday. Logitech also sells other Harmony remote models, but anything below the 620 was too little for what I needed and anything above is, frankly, overkill. The Harmony One, for example, costs almost twice as much and all it adds are a touch screen with color and a rechargeable battery. I like the idea of being able to recharge my remote, but it’s not worth the extra $79. I think the color touch screen is essentially a throwaway feature. I like my remote controls to have tactile feedback, and usually touch screens that small are terribly inaccurate. There’s also another model, the Harmony 1100, which is roughly the size of a sheet of paper and is almost entirely touchscreen-based, which means it would be even less useful to me than the Harmony One. I honestly don’t understand why somebody would purchase an accessory for their home theater that potentially costs more than the components themselves!

It was a snap to enter in the codes for all my devices, but I ran into a few snags:

  • My HTPC didn’t have an IR receiver, so I couldn’t control the Harmony with it.
  • The PlayStation 3 uses BlueTooth for its wireless devices, and doesn’t have an IR port, either.
  • The Wii doesn’t support IR.

The Wii turned out to be a moot point. You pretty much have to get up to get a Wiimote. If you turn on the Wii, you’re going to want to play a game, not watch a movie (Netflix rumors notwithstanding). So I added an activity (‘Play Wii’), and all it can do is turn on the TV and the receiver.

The PS3 presented more of a challenge. There are two items that could have worked here. Nyko makes a PS3 Remote, and Logitech sells a Blutooth->IR adapter. The Nyko device plugs into the PS3 via a USB port, which means it takes up on of the ports and can’t turn the unit on (because it draws its power from USB). The Logitech Harmony adapter actually translates IR commands to BlueTooth (meaning it can turn on the PS3), but it costs almost four times as much as the Nyko remote.

I finally decided to go with the Logitech one, because I wanted to keep a USB port free and for some reason the ability to turn the PS3 off and on was really, really important to me. Sometimes I’m obsessive about this sort of thing, and I decided I wanted as much control over my devices as possible. In hindsight, this purchase may have been the most impulsive one in my home theater-upgrading spree. Nevertheless, I ordered the thing and it worked right away. The most complicated thing I had to do was register it with the PlayStation 3’s OS.

With that problem solved, I turned to the matter of my HTPC. The bulk of my home theater viewing is done on this. I had search for something like this before, but couldn’t really find anything that matched what I wanted to do. Part of the problem is that my playback software of choice is MediaPortal, and lots of remotes didn’t work with it. Ideally, I wanted an IR receiver that could process remote commands and turn them into arbitrary keystrokes.

I don’t know if there’s anything that does precisely that, but I found something that’s close. It’s a PC remote that’s interpreted as a USB keyboard by your computer. I wasn’t 100% sure of what keystrokes the remote buttons translated into, but I saw enough (after doing some Google image searches) to realize that it included the four arrow keys, escape, and enter. These were the necessary keys for navigating the MediaPortal menus. The rest would be window dressing. And — bonus — somebody posted the remote’s codes to the Logitech database, which meant I would only have to do minimal tweaking to get it to work with MediaPortal. Another thing worth mentioning is that the PC remote itself has the ability to control the mouse. Unfortunately, it has its own pad for this and there’s no direct equivalent on the Harmony remote, but I still managed to get it programmed in, thanks to the customizable menus available on the Logitech remote.

Mapping the remote took some effort and a bit of ingenuity on my part. First I had to figure out what keys each of the PC remote’s buttons corresponded to. One approach would have been to open Notepad, press the remote buttons, and see what gets typed. I decided to do it another wayL I used another (unrelated) Logitech program. I sometimes use my HTPC to play emulators for old video game consoles, and had bought a couple of Logitech USB gamepads a while back. They came with a program that allows you to configure them. This program has a ‘record keystroke’ feature which tells you exactly which buttons are being pressed. So I can push each button on the PC remote in turn, and see on screen exactly what keystrokes are sent to the computer. This allowed me to easily map those same commands to the Harmony Remote’s buttons.

Interestingly, the commands I had the most trouble with were the transport buttons: play, pause, and stop. MediaPortal uses the space bar (in line with other media programs like VLC and QuickTime) to pause or play a video. This was not triggered directly by any button on the PC remote. The closest that remote came to pressing spacebar was via a clumsy numlock button, which turned the 0-9 digits into letters (kind of like a telephone pad). It was even worse for the stop functionality, which is triggered in MediaPortal by the ‘b’ key. As far as I could tell, I couldn’t reconfigure MediaPortal’s default controls to match one of the keycodes triggered by the PC remote.

But the remote did have four commands reserved for macros or program launching. It had A, B, C, and D buttons that triggered alt+shift+a, alt+shift+b,… and so on. So I downloaded AutoHotKey onto my HTPC. This is an open-source macro program that allows you to create scripts to control your PC automatically. I created a script that simply presses ‘B’, compiled it to an application (AutoHotKey can do that), created a shortcut to said application, and gave it the trigger alt+shift+a. Now the ‘A’ button on the PC remote (or its equivalent on the Harmony Remote) triggered this macro, which pressed ‘b’ and stopped whatever was playing via MediaPortal. Awesome!

I programmed the other hotkeys to do other things. ‘B’ presses the spacebar, which I assigned to the Harmony Remote’s play and pause buttons. ‘C’ opens MediaPortal itself (in case I boot the PC or otherwise need to run MediaPortal).

The most fun I had was programming the ‘D’ hotkey. I ran into a problem that I still haven’t been able to solve, but have developed a band-aid workaround. Sometimes, when I turn on the HTPC, it ‘forgets’ that it’s plugged into my TV via HDMI and no signal is sent. Luckily, I have another output from the HTPC to the TV via a VGA (or D-Sub) connector, and this one is never forgotten. So if I turn on the HTPC and for some reason it doesn’t know about the DVI port connected to the TV, I can use the VGA port and switch the monitors. But boy, what a pain! I wound up creating another AutoHotKey script that switches to the desktop, right-click it, chooses ‘Display Properties’, and tabs/toggles its way through the display dialog, changing over to the DVI output and applying the changes. It works like magic! If I turn on the HTPC and don’t get any video, I simply press the ‘D’ hotkey on the Harmony remote (which has been programmed into the customizable menu as ‘Fix Displays’), and everything is fixed!

I also programmed some other stuff into the Harmony remote’s custom menus. There are mouse control command, ctrl+esc for showing the start menu, Windows Key-D for showing the desktop, alt-tab for switching programs, and tab for configuring dialog boxes. The remote also had commands for my laptop (which came with an IR remote control) and a digital photo frame my parents had given my wife and me for our wedding. True, I don’t do much to control the frame, but it’s reassuring to know that I have the power.

This was the last step, and now I’m in complete control. Every command is at my fingertips. I had a total of nine devices and eight activities. I have One Remote to Rule Them All. I’m almost drunk off my own power. Too bad I can’t convince my wife to use the remote to control the PC (getting that to work I viewed as my crowning achievement) — she still uses the wireless keyboard.

Tags: , , , , , ,
Posted in Computing, Features, Media, Tech | No Comments »

Home Theater Without Breaking the Bank – Part 1

Monday, January 11th, 2010

A few months ago my home theater suffered a nearly-fatal blow: my receiver died. I was obligated to remove it and put in an old friend to hold down the fort until I could replace my fallen soldier. A couple of weeks ago I finally got a replacement, and started the process of planning, configuring, and upgrading my home theater. This weekend I put in the final piece of the puzzle, and can now say that I have a home theater to be proud of. And the best part is that I didn’t have to pay through the nose to get something I’m proud of. Until very recently, I worried I would have to build a whole new HTPC. Thankfully, a solution arose in the most unlikely of places.

With all of that work (some might even call it fun) out of the way, I can around to describing the thing, and hopefully sharing some tips so that others attempting something similar to what I did can get it done without quite as much hassle. This subject is kind of hard for me to write about, for two opposing reasons. The reason I am apt to write this all down is because I love technical things, and I especially love stereo (and home theater setup). However, I’m wary about writing about it because when I see others do so it often feels to me almost like bragging, as in, “look at how much neat stuff I have!” I realize that the fault probably lies with me and the way I perceive people talking about this stuff, but at the same time it makes me want to avoid that sort of thing, so I’ll try. I want to make this about the technology, not how much it costs. That’s why I’m focusing on the “without breaking the bank” part.

I’m going to divide this up into several articles, because I could write quite a lot about it. This first part will be about the solution I went with on my HTPC, and my search for a video player. I will also cover my remote control configuration and receiver setup later.

Part 1: Finding a Video Player

One of my chief goals was to find a way to play HD video without having to go too far out of my way. I knew this was likely to be a tough nut to crack. None of my computers are what you would call cutting-edge. I’m a web programmer, so I don’t really have to have a beefy CPU for compiling, and I’m not some knee-jerk gamer who dumps cash on the biggest, meanest graphics card. That means I’m never on the cutting edge of hardware: I wouldn’t utilize the bleeding edge anyway. At the same time, I realize that HD video is permeating many parts of our lives and it’s just going to become more prevalent, so I wanted to focus on future-proofing (or at least future-preparing) my setup.

My home theater is a modest Compaq box I got when I was in college and switched majors to Computer Science about five years ago. It has an AMD Athalon x64 processor, 2GB of RAM, and a decent video card I got a couple of years later (I think it’s an nVidia GeForce 7000 series card, and I know it has 128 MB RAM). I’ve got it hooked up to my receiver via a DVI-to-HDMI adapter and digital coaxial output. It can show 1080p (and that’s the way it runs), but I knew that it would take a miracle to get actual 1080p video running on it.

About a year and a half ago, I decided to use MediaPortal as the media library/playback app. I also looked at XBMC, but at the time it was too buggy to consider. There are yet other options, but MediaPortal is the one I settled on because of how configurable it is, and how suited it is to what I need (playback of video on Windows shares, and occasional music, too). I could have gone with a whole different Operating System, but settled on Windows XP as it is stable and lean compared to Vista. I considered Linux briefly, but decided that my hardware requirements would make getting it to work into a chore instead of a hobby. Things may have changed in the last eighteen months, but I’ve kind of settled upon Windows for now.

720p video worked right out the door via MediaPortal. I’m not 100% certain, but I’m confident that it wouldn’t play smoothly on this box without a little something called DVXA. DXVA is a Microsoft API that uses your graphics card to help decode demanding video (like H.264-encoded stuff, which is primarily what HD video uses these days). My confirmation of this is that the 720p video doesn’t run smoothly on this box in VLC, which as far as I know doesn’t use hardware acceleration at all.

1080p video flat out didn’t work. I figured it wouldn’t on this box. Betting on a long shot, I gave it a try anyway using both MediaPortal (which crashed) and Media Player Classic Home Cinema, which also uses DXVA but probably doesn’t have as much overhead as MediaPortal. No luck with either, which is exactly what I expected.

I decided to try out my laptop. It was marketed and sold as a ‘media center laptop.’ I got this one about two and a half years ago to replace an aging Dell Latitudes I’d inherited from my father. It came with Vista Home Premium (which I replaced with Business). It’s got a speedier CPU: a Core 2 Duo. It’s also got 4GB of RAM and an 8000-series GeForce. I figured this should be more than enough to play 1080p video.

Wrong! VLC couldn’t hack it. It looked like VLC didn’t even use use both cores while decompressing. I tried MPC HC and it sort of worked after I tweaked the settings. Some 1080p video, like animation, would play just fine. The beefiest files I threw at it worked, too. But other 1080p files which should have required less decoding muscle were stuttering and had terrible lip synch. In all other respects, the laptop would have been a golden solution: it had an HDMI output which carried both HD video and sound, and I had a free port on my receiver and a decent-sized cable. I finally managed to configure MPC HC to send the digital audio straight out over the HDMI instead of decoding it stereo, so I had 5.1 surround, too. But when I sat down to do a test run, it still had some problems. Video would stutter and lip synch would often drift, sometimes being correct and other times being just enough off to be frustrating.

At this point, I thought maybe my network was to blame. My server is downstairs in the basement. I had first tried a pair of Ethernet-over-power (also known as Powerlink) bridges, but I think my house’s wiring is too old and noisy to privde reliable throughput. It couldn’t even deliver reduced-size DVD rips from the server. I upgraded to a wireless bridge with 802.11n, and this was at least able to deliver a 4GB DVD rip in the time it took to watch it. I figured maybe my laptop was decoding video just fine, but not receiving content quickly enough. A quick copy of a 6.5 GB, hour-long 1080p video took about 43 minutes, which should have been enough to do it. But I tried watching the copied video direct from my hard drive, and it was still displaying the same behavior.

This is when things started to veer of into crazy land. I tried my wife’s PC, which is in the same room as the TV and has a slightly beefier processor and a much beefier graphics card than my laptop. It could play some video my laptop couldn’t, but bizarrely it couldn’t handle the video my laptop could.. At this point, I considered either building a new, extremely beefy HTPC or dropping some cash to run an HDMI cable the approximately 25 feet that would be necessary to connect my wife’s PC to the TV. Neither of these were ideal solutions, because I’d have to test a new 1080p video on both machines and screw around with display settings.

Strangely enough, my first-generation Intel iMac (just a Core Duo, not a Core 2) could play the same video my wife’s much newer PC could and could almost play the video that worked on my laptop — in VLC, no less. I had no idea why.

At this point I tried a couple of Linux LiveCDs on my laptop, figuring that Vista’s overhead may have been the problem, but ran into hardware configuration problems. This would also have required even more work than mentioned above. To watch 1080p video, I’d have to reboot with the LiveCD or install in onto my hard drive.

Is it clear I was close to madness at this point? Why could my 3.5 year-old iMac play most HD video just fine, while my newer laptop and my wife’s computer could only handle some of it? Why was MPC HC behaving so differently on different boxes when it was configured the same?

The solution came in the form of XBMC. I downloaded it on a whim, just to test it out. Not only could it play all my 1080p content at full resolution, but it could also do crazy things like fullscreen overlays and subtitles (just moving the mouse was enough to make playback jittery in MPC HC)! I even got it configured to output 5.1 digital audio without too much fiddling. So far, it’s played everything I’ve thrown at it.

The only problem I’ve seen is in some 1080 buffering issues. I think I know why. Today’s video codecs don’t encode information at the same bitrate all the time. A file has an average bitrate, which can roughly be calculated by dividing its size by its length. But the bitrate varies considerably. From 1:00:00 1:01:00, for example, it might play at 2 Mb / sec, but from 1:01:00 to 1:02:00, it might play at 3 Mb / sec or more. A smart codec squeezes as much extraneous data out of a scene as it can. This is easy in scenes with little motion. But a scene with a lot of action might have a much higher instantaneous bitrate. So even though my wireless N network can download the whole video in less time than is required to play it, it could be that certain parts have such a high bitrate that they can’t be played at my network’s transmission speed, while most of the rest of the video falls well below the average.

There’s not much I can do about this, aside from ‘pre-buffering’ HD video by copying it to the laptop first. I can’t lay cable in my house because it’s rented. My wife wouldn’t approve of stringing long ethernet cables down the stairs, either. Maybe there’s an option to increase XBMC’s video buffer. It definitely seems smarter than, for example, Amazon’s video-on-demand buffer or YouTube, but it’s hard to gauge these things. I might wind up getting a big hard drive exclusively for my HD content, attached to the network upstairs. Maybe I’ll look into a NAS solution.

So, here’s an overview of what I use the computers for:

  • HTPC Box — SD video, DVD rips, HD video up to 720p. Also, I run various emulators off it, and play music from my server. This is all done via MediaPortal. Video is via HDMI out, audio is via digital coax up to 5.1 surround.
  • Laptop — 1080p video played in XBMC. Video and audio via HDMI, up to 5.1 surround.

Not so complicated when it’s put that simply. The best part is that I didn’t have to build (and pay for) a new computer just to get full HD playback. The only drawback is that I can’t do anything else (like browse the web) on that laptop. I’ve still got the old Dell Latitude, so I could do it that way, but usually if I’m watching 1080p video, it’s got my full attention.

For Part II, I’ll go through what it took to get everything controlled by one remote.

Tags: , , , , , , ,
Posted in Computing, Features, Media, Tech | No Comments »

Music Section is Live

Friday, October 23rd, 2009

After months of having a tantalizing-looking ‘Music’ link in this site’s main navigation but no payoff after the click, I’m happy to tell the world that the music section is live. There isn’t too much up now, aside from some free MP3s and an archive of music news, but there will be more content when I have more time. I would like to include some info about the lovely Why?-Fi Studio and how I record a song, but those will require a lot of work. I also plan to have an ‘upcoming albums’ section, but that might be unrealistic, as I would feel obligated to give all the projects I’m working on actual deadlines, then would feel the need to meet those deadlines. You see my dilemma.

Someday I’ll have an online store. The plan includes physical copies of CDs as well as digital downloads, initially in mp3 but probably in FLAC eventually. What’s going to be neat is that the digital downloads will feature a ‘Pay What You Like’ option (yeah, yeah, “Simpsons Radiohead did it”), which will go all the way down to free if that’s what people want to pay. I’m having a little trouble with the logistics of the whole project because I would like to include cover songs (I’ve recorded a lot of cover songs, my Christmas albums are almost all covers, and even The Suckers recorded a full-album cover of the Ramones’ first record), but I have to deal with the nasty reality of licensing, which looks to be as much negative fun as making the songs in the first place was. So the covers are going to have a minimum price so I don’t wind up losing money on the venture.

Tags: , , ,
Posted in Music, Recording, Recording News, Tech | No Comments »

Organizing The Music

Monday, October 12th, 2009

So I’ve begun the process of manually merging my music collection. It’s a mess, quite frankly. I’ve got MP3s I’ve ripped or purchased on four different computers, spread throughout many directories. Compounding this is my iPod, which usually carries the latest tracks that I’ve added. Here’s how I’m organizing things. The fun part is that I got to write a Python script to help out.

First Steps

I’ve got one folder that was my primary music folder throughout my time in school. It rests on my file server. It generally contains all my music and is the most authoratative ‘source.’ In addition, it was the initial source, the ‘seed’ if you will, for the tracks on the iPod. At one point in the distant past, my iPod contained the tracks from this folder and nothing else. This is what I’m going to start with. To really drive home the point of my fresh start, I creates a share on my file server and started anew. These tracks wound up in a folder called ‘library.’

This is already a good start. I’ve been pretty meticulous in organizing my music library, essentially by artist then by album. The /library/ folder is going to be my new, massively-integrated library, as soon as I get finished organzing.

The iPod

Since my iPod contains several albums that never made it to the music share for one reason or another, it can also be considered ‘authoratative.’ So I ripped its contents to another folder in the new music shared, called /iPod/. I used the excellent tool SharePod to do this, as it allowed me to rip the tracks to artist/album folders with very little hastle.

Other Sources

I then rounded up all my other music, and put it into an ‘unsorted’ directory. This is stuff I would go through item by item, once the two primary sources were sorted out, and include or not include depending on if it wound up on my iPod or not. I have yet to get all the way through this step.

The Script

This is the important bit. I wrote a Python script to crawl through the two directories in parallel, and note any missing files or directories. This way, I’ll know what I need to copy from the /iPod/ folder to the /library/ folder. It’s a fairly simple command-line script, used like this:

compare.py left right outfile [filter1,filter2...]

left is the first directory, right is the second. outfile is a text file that the differences will be written to, and the [filter]s allow me to specify a whitelist of file types I care about. In this case, the whitelist would be restricted to audio file types. Here is the command I wound up running (drive Y:\ is the share I set up):

compare.ph Y:\library\ Y:\iPod\ Y:\results.txt mp3,m4a

This ran the Python script, comparing the /library/ and /iPod/ directories (and, recursively, their children), saving the log of all the differences to results.txt at the root of the share. Additionally, the program ignored any files except mp3 or m4a files (and directories, obviously). I wound up with a list of all the folders and files unique to the initial library and the one copied from my iPod. Then it was a simple matter to copy the iPod-unique folders to the library. I could even use it to update my iPod if I really wanted to, although it’s running pretty close to full now.

Of course, there’s still a lot of work to do: I’ve got to tag the /unsorted/ files. Have I mentioned how meticulous I am about my music library?

Source Code

import os # for files and paths
import sys # for command line arguments
 
def matches (path, fileName, filter):
    """Returns true if the given file matches the filter or is a directory, false otherwise.
    path - the directory the file resides in
    fileName - the name of the file in question
    filter - Either None to indicate no filtering should be applied, or a list of allowed extensions."""
    if filter == None:
        return True
    else:
        # if it's a directory, return true
        if (os.path.isdir(os.path.join(path, fileName))):
            return True
        ext = fileName.split(".").pop()
        return (ext in filter)
 
 
def compareDirectories (leftPath, rightPath, uniqueLeft, uniqueRight, filter = None):
    """Recursive function to compare the contents of two given directories. Two lists are
supplied to keep track of the unique files. An optional filter is allowed.
    leftPath - The path to the first directory.
    rightPath - The path to the second directory.
    uniqueLeft - A master list of files unique to the left directory tree.
    uniqueRight - A master list of files unique to the right directory tree.
    filter - Either None, or a list of allowed (whitelist) extensions for files. A unique file in
            either the left or right directory will not be counted as unique if its extension
            does not match one of the filter items."""
 
    # get contents of directories
    left = sorted(os.listdir(leftPath));
    right = sorted(os.listdir(rightPath));
 
    # without a filter, just add all unique files
    if (filter == None):
        # append unique files by using a list comprehension to get all files on one side
        # that are not on the other side
        uniqueLeft[len(uniqueLeft):] = [os.path.join(rightPath, fileName) for fileName in right if fileName not in left]
        uniqueRight[len(uniqueRight):] = [os.path.join(leftPath, fileName) for fileName in left if fileName not in right]
    # otherwise, use the filter function
    else:
        # same as above, but also checks to see that the files match the given filters
        uniqueLeft[len(uniqueLeft):] = [os.path.join(rightPath, fileName) for fileName in right
                                        if fileName not in left and matches(rightPath, fileName, filter)]
        uniqueRight[len(uniqueRight):] = [os.path.join(leftPath, fileName) for fileName in left
                                          if fileName not in right and matches(leftPath, fileName, filter)]
 
    # get a list of files in both directores. Since they by definition must be in both,
    # we can pull them from either side using a list comprehension to check that they're
    # in the other.
    both = [fileName for fileName in left if fileName in right]
 
    # now go through and recursively call the function for any directories in both parent directories
    for fileName in both:
        leftChild = os.path.join(leftPath, fileName)
        rightChild = os.path.join(rightPath, fileName)
        if (os.path.isdir(leftChild) and os.path.isdir(rightChild)):
            compareDirectories(leftChild, rightChild, uniqueLeft, uniqueRight, filter)
 
def usage ():
    print "\n\ncompare.py"
    print "Compares two directories recursively and lists files or folders unique to each one.\n"
    print "compare.py left right outfile [filter1,filter2...]"
    print "\tleft\tFirst directory to compare"
    print "\tright\tSecond directory to compare"
    print "\toutfile\tText file that results are written to"
    print "\t[filter1,filter2]\tOptional comma-separated whitelist"
    print" \t\t\t\tof extensions for files"
    exit()
 
if __name__ == "__main__":
    # slice off name of program from args
    args = sys.argv[1:]
 
    # if there's an incorrect number of parameters, print the usage
    if len(args) < 3 or len(args) > 4:
        usage()
 
    # set up filter whitelist, if any
    filter = None
    if len(args) == 4:
        filter = args[3].split(",")
 
    # set up lists of unique files on both sides
    uniqueRight = list();
    uniqueLeft = list();
 
    # do the comparison recursively
    compareDirectories(args[0], args[1], uniqueLeft, uniqueRight, filter)
 
    # write to the file
    out = open(args[2], 'w')
 
    out.write("UNIQUE TO LEFT:\n")
    for fileName in uniqueLeft:
        out.write(fileName + "\n")
 
    out.write("\nUNIQUE TO RIGHT:\n")
    for fileName in uniqueRight:
       out.write(fileName + "\n")
 
    out.close()

Tags: , , , , ,
Posted in Computing, Media, Music, Programming | No Comments »

Wherin Crackers Strike

Monday, October 12th, 2009

Sometimes, even the best of us can get a good lesson in security.

Last week, I found out that my website was attacked by crackers. Notice I use the word ‘crackers,’ not ‘hackers’, because hackers are not crackers, and it’s important to maintain the distinction. From what they left I can tell that they clearly fall in the black-hat camp. If I’d gotten a warning e-mail or a message on my site to tighten my security, I would take it as a reminder to batten down the hatches. But since they just left a juicy payload, I can assume that they’re up to no good.

From looking at the files they left, I can tell they wanted continuing access to the shell account on my web host, and they wanted to do so in secret. Since I work in web programming, I’ve seen my share of more-conspicuous payloads. These are usually surreptitious JavaScript files, plopped at the end of legit PHP scripts to do nasty things. Most of what I’ve seen have been little snippets of code that act as drive-by downloaders, trying to pull malicious executables onto hapless users’ computers. The only thing that my attacker’s payload did was grant PHP execution and shell access. Not damaging to anyone who happens across a compromised site, but potentially damanging to me — it essentially gave them free reign of my SHH account.

I could speculate about what they wanted to do. There are a lot of ways to do nasty things on the web. PHP is web-aware enough to allow them to do as they please and ( assuming they’ve covered their track properly) not get caught, either. But I doubt they were able to get much nastiness accomplished, because they executed their attack sloppily: they dropped their payload in the wrong directory. They put their files one level above my web root, meaning that the scripts were inaccessible over the web. At first I thought they may have found an exploit in my framework to allow access to anything on the filesystem, but after reviewing their code, I can see that this is not the case. I guess they wanted to get in, drop off the files, and get out. They may have even made a second attempt after determining that the first one didn’t work; I found two files with exactly the same code.

How did they do it? I’m not certain, but I have an idea. The only web app I use is WordPress, and I’m updated to the current version, so this is an unlikely point-of-entry. They would have to know about an exploit that’s not been reported yet, which is possible, but doubtful.

Much more likely is that they managed to guess or sniff my password. I’m the guilty one here, as I was using a simple password that I’ve been using for years, which had little variation, was dictionary-based, and was much too short. In addition to that, I’ve got a webcam at home that posts images fairly frequently (at regular intervals), and it used the same account as my main FTP/Shell account. As you may know, FTP passwords are sent in cleartext, so this was definitely a potential point of entry. Assuming that the password was the point of failure, I’m lucky that they didn’t do more damage, as I used the same password for shell access, MySQL, and even my web control panel, so they theoretically could have locked me out of everything. I’m hypothesizing here, but I’d guess such an attack would be counterproductive; I think they just wanted another remote-control node on the web to carry out any dirty business they happened to think up.

Of course, I took steps to ensure that things are more locked down, starting with changing every password associated with this site. I did this as soon as I found out, and before anything else, to sever any venues they might have had to retaliate against me. Then I checked WordPress for updates, just in case there might have been an exploit I missed. Next, I updated how my webcam saves the periodic images and created a new account specifically for it. Finally, I did a quick review of my code base, making sure they hadn’t left another way to re-gain access. Basically, I pulled down my whole site and did a global search for any of the crackers’ friend functions: eval(), the base64 functions, system() and friends, and file-related functions. I’ve still got to re-upload all the code to feel 100% safe again, but I’m pretty certain that nothing slipped by.

Stay tuned, because after I’ve further reviewed what they left and when I’ve done a bit more research, I’ll post an analysis of the code itself.

Tags: , , , ,
Posted in Computing, Programming | No Comments »

How I Survived My Net Outage

Monday, August 17th, 2009

There are about fifteen points of failure between my upstairs computer and the Internet at large. So when my web access stopped working last Thursday night, I had a long way to go before I blamed my Internet Service Provider (Bresnan). Having worked at a sort of ISP for four years (the DirectConnect Office at the University of Montana), I was quite used to customers immediately blaming their provider, instead of the technology that was immediately at their control. After all, they hadn’t changed anything, right, so it must be the connection itself. Or, as they each usually liked to call it, “my Internet.” Oh, that’s nice. Got your own personal Internet, huh?

The point is, I didn’t call my ISP the second something went wrong. Because the probability that it was on my end was greater than 86.525%, I started troubleshooting my way upstream. Now, between my upstairs computer, there are a number of failure points. To wit:

  1. The computer itself connects to a D-Link wireless bridge (the model number escapes me now). This bridge works flawlessly until something goes wrong, at which point it acts like a brick with a five-port switch.
  2. Downstairs is the bridge’s counterpart, a D-Link Draft N access point. Because of the layout of my house and probably due to its age, the highest signal strength I can on the wireless bridge upstairs is about 66%. The signal has to travel through two walls and a ceiling to get upstairs, and I have the strength set lower than maximum to keep the signal within a reasonable range of my house.
  3. From there, the signal has to go through a switch to the actual router, which is a LinkSys model running the Tomato firmware.
  4. Finally, from the router, we get to the cable modem, which plugs into the ‘real’ Internet and signals the end of my responsibility.

Between each of these nodes, of course, are the usual points of failure. This includes the hodge-podge of Ethernet cables I’ve collected throughout my life, any of which might have lost the tab that’s supposed to hold it in place. I had a bit of troubleshooting to do, but was still pretty certain that the problem rested with me.

The first thing I did was connect to the router’s web configuration. This worked right away, which eliminated most of my signal chain. Unless my connection problems were the result of some bad Voodoo (which has happened before; maybe sometime I’ll tell you the horror story of my Ethernet-Over-Power attempts to link my upstairs and downstairs network legs), it was looking more and more like Bresnan’s fault. This was confirmed when I checked the lights on my cable modem. Being a rational-thinking person, I resisted the temptation to reset my router and my cable modem. I have known people who instantly do this even if their connection is a bit slow, and I assume that the one or two times this actually worked was enough to reinforce the superstition in them. The final nail in coffin for Bresnan was that I could release and renew my router’s IP address, but could not use my router to ping anything upstream, except my router’s router.

Since it was a bit late in the day, I resisted the urge to call my ISP and complain. Often people would aggravate me by calling to complain about outages at school. These outages could be beyond our control, as we were not the people in charge of the ‘pipe’ to the Internet, but were merely intermediaries between the campus’s IT department and the dorm residents. I figured that if it were a serious problem, my connection would still not be working in the morning.

As the sun was climbing in the sky I checked, and everything was working. I really didn’t give it much more thought; after all six hours of downtime in the fourteen months I’d been a customer translated to at least three nines (99.9%) of uptime, which was pretty good for a residential service.

I didn’t give it much thought, but my ISP did.

I’m used to being given a bad shake by corporations, but this time Bresnan came through. This weekend, I got a voicemail from an unknown number, which turned out to be a recording from them, apologizing for my downtime. This wasn’t a half-assed CYA thing, because it had obviously cost them some money: somebody had to write the words, and they had to call all the customers who were affected, which would have cost them a bit. I wasn’t expecting any sort of acknowledgement of the outage; after all, I hadn’t called to complain. The fact that they left me a message means that they left everyone who was affected a message, too. And that shows that they care. This is good for me, because a company that cares about its image is probably less likely to be a jerk to its customers.

Often in business, those who are loudest are the ones who receive the most attention. It’s good to see that every once in a corporations can rise about the soullessness with which we often endow them, put on a human face, and treat their customers with respect.

Tags: , , , ,
Posted in Tech | 1 Comment »

Streaming Video: A Rant From Twitter

Friday, August 7th, 2009

Why can’t streaming video player programmers EVER get their buffer size prediction algorithms correct? Or, failing that, just make them more fault-tolerant so they buffer more than is necessary to start playing? I’m watching Amazon video on demand, and golly gee, it would be nice if I had control to buffer as much as I wanted, instead of what the moronic algorithm programmed into it thinks is enough.

Other irksome things: When it has to re-buffer, it takes away the progress indicator. This is because during 33% of rebuffers, the connection cuts. This way, I have to guess where I was on the movie when I re-load the page. Amazon, I know you’re trying to idiot-proof your software, but can you please put up something so we have some semblance of control? All I want is a truthful indicator of the connection status, and control over the buffer… is that too much to ask?

Oh, one more thing: If I pause, that’s your opportunity to LOAD AS MUCH BUFFER AS POSSIBLE. I’m hoping some progress has been made since this rant started, but I highly doubt that.

Tags: , ,
Posted in Computing, Media, Rants | No Comments »

Bad Vista UI! Bad!

Monday, February 11th, 2008

We all get to a point where our open window account in Explorer skyrockets. I use the ‘Run’ dialog to get to a lot of places, which opens each directory in a new window. So I often find myself with all my Explorer windows grouped together on the taskbar, and something like 16 or more directories in the listing. It’s natural to just want to ‘reset’ things and close them all.

But beware! The Explorer group includes not only your directory windows, but also any copy jobs going on. If you right-click and choose ‘Close Group’ when you are copying, your job in unceremoniously canceled (which itself could take a while, depending on how cranky the Vista copy routines are that day). God help you if you happen to be copying many gigabytes of files.

This is bad user interface design. Directory windows and copy windows should not be grouped together. A directory window displays information, while a copy windows displays a process. Yes, they run in the same program (Explorer), but they are two absolutely different things, and should be grouped together into two separate groups on the taskbar. A less-correct alternative, but probably easier to implement in software, would be to close all the directory windows, then ask the user — for each copy job — whether or not the copy should be canceled.

Tags: , , , ,
Posted in Computing | No Comments »

Wickedpistia

Tuesday, February 6th, 2007

I constantly find reasons never, ever to trust technology enough to become an astronaut.

Reason #1: The iPod-cum-brick. Today, there was an Adobe User Group meeting. On the way across the oval (which has recently become an uncrossable sea thanks to constantly freezing and melting ice sheets which once were snow. It’s pretty neat, because your shoes simultaneously come into contact with 1) water 2) ice and 3) mud, which means that you can get your shoes muddy, soak your feet, and fall on your ass, all at the same time!), I was listening to it just fine. Full charge, no problems, no skipping, nada. I put it away for the meeting. On the way back, the damn thing wouldn’t do anything! No apple screen, no iPod-sticking-his-tongue-out, nothing. I tried to fix it at home, first plugging it into its power adapter. No dice. Then my laptop’s USB. Still, nothing. Reset, reset, reset — zilch. The ‘5 R’s’ yielded no results. So now I apparently have a dead iPod. The worst part is that this isn’t the first time this has happened! About a month ago, I actually had to call tech support. For some reason (and this is before I got through, so the tech-guy-gadget-fixing-auro wasn’t in effect yet), on my twentieth attempt at restarting it (hold-on, hold-off, Menu and Center pressed and held together), it started working. Oh, yeah, and this isn’t even my first iPod! My first one died one day for similarly inexplicable reasons. Gee, Apple, you’d think for a grand total of $650 dollars I could possibly not by an unreliable piece of crap… twice.

Reason #2: Retarded torrents. For some reason (possibly the alignment of the moons), every time I’m downloading sweet TV shows via BitTorrent, nothing works. Usually, setting my client’s encryption to forced or enabled (whichever it currently is not set to) cures the problem. Not tonight. I tried four or five times, then snagged a torrent I knew would have seeds, all to no avail. So I got started on Reason #3 (see below) and came back to it after half an hour when — voilà — it started downloading. Of course, my episode of Heroes was going at 100 k/sec last night, but now, with only a third left, it was going at 20 k/sec, despite having the same number of seeds.

Don’t you love that? It seems that, regardless of your method of illicit p2p download (BitTorrent, Gnutella, even ancient Napster), you always wind up having 5 minutes left on your download for at least an hour, often more time. I assume my seeders are all d-bags who coordinate their efforts to frustrate me just enough so that I come back for next week’s episode.

Reason #3: Tried to write a paper about Python (the language, not the aeronautical beast). Finished it. Tried to upload it with the shitty Blackboard upload applet (that’s right! Start the JVM to accomplish something that can be done with a plain old HTML form!). Guess what? FireFox crashed! Tried in IE — now the whole box crashes! And I’m not running a bunch of crap software, as far as I know. After the restart, it went right up. But I found it amusing that submitting the paper took about 3% of the entire time spent on the damn thing.

Wait. It wasn’t amusing. It PISSED ME OFF.

By the way, while writing this I must have clicked out of the form text area and tried to delete something, because I hit backspace and immediately navigated away from the page. My blood boiled for half a second as I realized I might have lost this entire rant. To Blogger’s credit, it did warn me. But I’m so used to irritating popup messages that I typically click through familiar ones without thinking them through. Thankfully, I’m paranoid enough to copy and paste (
just did it) after ever sentence as a poor man’s save.

And I was going to try to install Windows Vista on my computer tonight. With my tech karma right now, the setup would probably error out so immensely, so enormously that I’d wind up reformatting my Mac’s hard drive, too.

Tags: , , , , , ,
Posted in Computing, Rants, School | No Comments »

Nerd Score

Sunday, November 5th, 2006

I am nerdier than 93% of all people. Are you nerdier? Click here to find out!

I don’t know whether or not to be proud of this…

Tags: ,
Posted in Miscellaneous, Tech | No Comments »

« Older Entries