DFW Pythoneers Meeting June 8, 2013

16 Python Enthusiasts showed up at the Gravity Center for the meeting. This will be the last meeting at the Gravity Center, a new meeting space will need to be found.

We discussed Python and other resources for young programmers. John discussed Cocos2d, Greg demoed Python koans, and Jeremy show how to install Pelican and deploy a static site on Amazon S3.

DFW Pythoneers Meeting May 14, 2013

I like to talk about Python. - Kevin Horn

11 Python Enthusiasts showed up at the Gravity Center for the meeting. After brief introductions, Kevin was gracious enough to take charge of the presentation and demo vitualenv and pip.

Kevin also discussed various package repositories, Unofficial Windows Binaries for Python Extension Packages, scipy, numpy, ipython, mingw and Chocolatey for Windows.

Mention was made of the forth coming PyTexas conference in August and Pyvideo.org, where various talks and tutorial are available for view.

The main focus of the second half of the meeting was data, databases and Python. Topics of discussion were: CSV, json, pickle, dbapi, SQLite, SQLAlchemy, sqlautocode and Alembic.

After the meeting a group of us went to Cafe Brazil for food and further discussion.

Netflix and Python

There's a nice post on their technical blog about how Netflix uses Python.

I found this section quite interesting:

Data Science and Engineering

Our Data Science and Engineering teams rely heavily on Python to help surface insights from the vast quantities of data produced by the organization. Python is used in tools for monitoring data quality, managing data movement and syncing, expressing business logic inside our ETL workflows, and running various web applications to visualize data.

One such application is Sting, a lightweight RESTful web service that slices, dices, and produces visualizations of large in-memory datasets. Our data science teams use Sting to analyze and iterate against the results of Hive queries on our big data platform. While a Hive query may take hours to complete, once the initial dataset is loaded in Sting, additional iterations using OLAP style operations enjoy sub-second response times. Datasets can be set to periodically refresh, so results are kept fresh and up to date. Sting is written entirely in Python, making heavy use of libraries such as pandas and numpy to perform fast filtering and aggregation operations.

Here's the video from PyCon 2013: http://pyvideo.org/video/1743/python-at-netflix

Text processing: The Bottom Line

You get all types of data formats when you deal with clients and financial data. Some send you nicely delimited text files with an current data dictionary. Some send Excel files that look like the intern's preschooler designed them. But sometimes you end up with a report consisting of pages of fixed-width text designed to be print off on the green-bar paper printer by the office AS/400.

If you need assistance in parsing text files, you can use commercial applications designed to handle the job like Monarch. There's also many tools and utilities designed to view and parse text files. Both Scott Hanselman and Buck Woody have detailed lists that you should peruse and explore.
But let's our skills and tackle the problem programatically.

The nice thing about many of the fixed-width text reports is they are very consistent in layout and organization, making them easy to parse.  If they are generated from an accounting system that includes the GL, (General Ledger), account number on each row, then you probably have the key to pulling out the information needed on a periodical basis. Let's see an small example.

Federal Borrowings Program

Notice the layout is very regimented with nicely formatted columns, descriptive headers and unique account numbers. The normal way a novice handles this type of file is to hand edit it and then try to clean up the result in Excel. (Shudder!) This report's organization makes it easy to write a simple utility rip the needed values. Even if the file is in a printable "report" format with headers on each page, it's a simple task to ignore these rows by focusing on the ACCOUNT column.

Sometimes you don't need every row since you don't want to load the data back into a database, you want to pull out specific totals and sub totals. It's easy enough to feed a list of account numbers or GL items to a routine, along with a list of position and widths of the account/items and the position and widths of the balances. You then end up with a dictionary, (Python, C#), a data structure that you can reference for calcutions or export/return to be handled by another process.  The process is something like this:

1. Pass file name and list of items to a routine
2. Create a dictionary structure with the list of items as the key values
3. Read in each line of the file, looking for matching keys, (using position and width)
4. If match found, populate the value for the matching key, (using position and width)
5. Continue till done with file.
6. Export dictionary to files, do calulations, or whatever.

Note: If you intend to do calculations on the values and wish to use them as numeric values, you will convert the text to numeric. This means you will probably have to clean up the currency characters and thousands separators. Easy to do in Python, but sometimes tricky in C#. In the case of C# include:  using System.Globalization;

Then use the following method:

Keep it Clean

As a developer, I was paid the ultimate complement by a coworker last week.

"Hey, that PowerShell script you wrote is really clean." he said in passing.

This was a script that I had ported from another language, then tweaked and forwarded to him to help manage some server resources. It was an ugly hack originally intended to solve a personal need, but I re-factored it to make it more modular, simpler, readable and added a few key comments.

The main benefit to clean code is that it's easy to come back months later and modified to suit your needs without having to do some sort of digital archaeology. But the best outcome is when others can use it and maintain it without any additional assistance from yourself.

Would you like some Data with that?

 Water, water, everywhere, Nor any drop to drink.

One thing that always bugged me about many of the talks and conferences I go to is the lack of good real world datasets and examples. There is the ubiquitous use of Adventureworks, which does fine for many demonstrations. Or the session based on the presenters experience with his employers assets, which are not accessible to the audience to view or play with. And there's the MVP speech with the sports statistics and the matching ball cap discussing ERA or passer efficiency rating to audiences from other cultures that follow different sports. And there's vendors that offer tools to generate sanitized datasets. If you need small or large datasets hopefully you don't always resort to these fall-backs since there are terabytes of interesting public data available on the Internet

Open or "public" data as it is called as been around for years. Before the WWW was in the public spotlight, you could order various data sets and source code on physical media from vendors. Two decades later, with the acceptance of the Internet and the increase of bandwidth, there's a plethora of sources of a huge variety of data sets available. One good stopping point for an overview is Data.Gov, an aggregate of Open Federal Data sources and tools.

Before you dive in and start grabbing collections of miscellaneous agricultural and health care stats from online sources, you need to have a idea of what type, quality and quantity of data set you are seeking. It's probably better to pick a domain that you have an understanding and experience in. And it doesn't hurt to select a data set that may solve a personal itch or business problem.

What's the Frequency, Kenneth?

One of my favourite online databases to pull from is the FCC ULS database. The FCC, (Federal Communication Commission), is responsible from managing the RF, (radio waves) and other communication in the United States. The ULS (Universal Licensing System) is a system to keep track of licenses, frequencency allocations and other business related to the FCC. As an amateur radio license holder, it's a fun to keep track of my and several hundred thousands of other "ham" license holders. As a database professional it's a open, well documented source of real world addresses with which to test skills, geocoding and CASS certification. So let's grab the Amateur Radio Service License database.

The license database (l_mat.zip) is an archive over 400 MBs in size when expanded, so make sure you have the resources to handle it. Once you have the data extracted, it's time to take inventory and break out the tool kit.

Pragmatism vs Partisanship

You ate Chinese food, so obviously you must hate Europeans...

Sounds silly doesn't it? So was the type of reaction I got from a data professional when I showed him a new book on data analysis that I was excited to add to my library. The software language didn't match his worldview or career investment, so I was labeled a "Microsoft basher".  Which is silly since we were at event for users of Microsoft software, I was using a Windows phone and two out of the three operating system I was running on my laptop were Windows 7 and Windows Server 2012.  And I spent much of the time taking notes in OneNote and discussing PowerShell 3 and SQL Server 12 with my cohort.

And the irony of situation is that Microsoft and many of it's employees and advocates recognized that not all the great tools and goodness flows from the mother-ship in Redmond. Buck Woody, a author and well known Microsoft database and Azure evangelist recommends installing OSS text-handling utilities when setting up your Data Science Laboratory. Another well known Microsoft technologist, Scott Hanselman, suggests many third party tools and has a recent post discussing GitHub and line endings. With the existence of CodePlex,the inclusion of Git support  in Visual Studio and offering Linux VMs on Azure, Microsoft is becoming more pragmatic and inclusive in regards to OSS.

And OSS has growing garnering commercial support. Red Hat has been making money for years. VMware supports both commercial and OSS hosts and guest. Some of the projects on CodePlex get adopted by commercial companies. And data analysis tools featured in the book that seed of this post have commercial support from a company, Continuum Analytics, which just received a grant from DARPA, to further develop their tools.

So, while disappointed in the reaction I received from this individual, I still respect him and hope to demonstrate the power of using both OSS and Microsoft tools together to tackle some tough data problems.