> python concordance.py '.{30}(?i)reactor.{30}' fukushima.txt
arch 2011 of the four damaged reactor buildings Date 11 March 2011
es six separate boiling water reactors originally designed by Gener
O). At the time of the quake, Reactor 4 had been de-fueled while 5
the earthquake, the remaining reactors 1-3 shut down automatically
olant water through a nuclear reactor for several days in order to
wn. As the pumps stopped, the reactors overheated due to the normal
 first few days after nuclear reactor shutdown (smaller amounts of
, only prompt flooding of the reactors with seawater could have coo
ause it would ruin the costly reactors permanently. Flooding with s
 the water boiled away in the reactors and the water levels in the
...

The Python code that generates this information is short and simple. There are two command line arguments: the expression used to search the text and the name of the text file. The Python program reads the input text, then uses the findall method from the re library to generate a list of matches. A simple for loop iterates over the results and prints them. Here is the core of the program:

import re
import sys
fp = open(sys.argv[2])
txt = fp.read()
fp.close()
for matchstr in re.findall(sys.argv[1],txt):
    print matchstr

This script is short because most of the processing is happening in the re.findall function when evaluating the regular expression. Using this powerful capability, you can create a wide variety of input expressions to match the content in your text file. Here are just a few examples:

At first glance, regular expressions might seem bewildering and incomprehensible. But creating a simple template with examples can help you towards the productive use of regular expressions. Another issue is running a Python program on the command line, which is not a common use model on a Windows computer in 2013. In the next post I’ll explore how to make this powerful script more user-friendly and accessible.

One way that scripted text processing can help your documentation is to identify inconsistencies and style problems in your writing. In my first assignment as a Technical Writer, I was handed a user guide and told to “clean it up.” Documents that are passed from writer to writer tend to gather cruft over time and inherit different writing styles from different writers. How do you get a quantitative understanding of the update task, other than reading the document and noting the problems one-by-one? To answer this question, I wrote a concordance generator.

A concordance is a list of words used in a body of work, with their immediate contexts.
[http://en.wikipedia.org/wiki/Concordance_(publishing)]. Using a concordance, you can view word usage in context and quickly spot inconsistencies. I’ve found that this technique is much, much more effective than using the word processor’s search feature to look through the document (although the latest version of Microsoft Word does contain improvements in this area.)

As an example, I’ll use the text from a random Wikipedia article. I chose the page describing the Fukushima nuclear disaster at http://en.wikipedia.org/wiki/Fukushima_Daiichi_nuclear_disaster and saved the text into a file called fukushima.txt. Using my concordance tool, I first examined how the word “reactor” was used in the text:

> python concordance.py '.{30}(?i)reactor.{30}' fukushima.txt
arch 2011 of the four damaged reactor buildings Date 11 March 2011
es six separate boiling water reactors originally designed by Gener
O). At the time of the quake, Reactor 4 had been de-fueled while 5
the earthquake, the remaining reactors 1-3 shut down automatically
olant water through a nuclear reactor for several days in order to
wn. As the pumps stopped, the reactors overheated due to the normal
 first few days after nuclear reactor shutdown (smaller amounts of
, only prompt flooding of the reactors with seawater could have coo
ause it would ruin the costly reactors permanently. Flooding with s
 the water boiled away in the reactors and the water levels in the
...

The concordance.py file contains the actual program script which we’ll look at in a later post. The arguments to the program are the match string and a list of files to process. The string is a regular expression that is applied to the input text. If you’ve never seen a regular expression, it can be a bit intimidating. However, if you stick to a few basic patterns, you’d be amazed at what you can accomplish. The ".{30}" element says “match any character (.) thirty times.” This provides context by matching any thirty characters before and any thirty characters after the substring of interest. The "(?i)" element tells the matcher to ignore case sensitivity. The program reads the fukushima.txt file and prints out every match for the expression you provide.

Here’s a more interesting example. I want to see how the terms “shutdown” and “shut down” are used in context. In a typical word processor, I would probably search for “shut” to make sure I catch both cases. Using concordance.py, I can modify my search expression to capture exactly what I want:

> python concordance.py '.{30}(?i)shut.?down.{30}' fukushima.txt
ed while 5 and 6 were in cold shutdown for planned maintenance.[8] I
e, the remaining reactors 1-3 shut down automatically and emergency g
from melting down after being shut down. As the pumps stopped, the re
ew days after nuclear reactor shutdown (smaller amounts of this heat
...

The "shut.?down" expression searches for “shut”, followed optionally by any character, followed by “down”. This term matches the strings “shut down”, “shut-down”, and “shutdown”. If you’re a skeptic, you’re probably thinking “Meh, I could just search for ‘shut’ and find all these cases anyway.”

Here’s a trickier example. The Fukushima plant involved a number of different buildings or “units” which contained reactors. Your first thought might be to search for “unit”:

> python concordance.py '.{30}(?i)unit.{30}' fukushima.txt
...
tance of nuclear power in the United States was eroded sharply f
 Switzerland, Taiwan, and the United States. Much of the help an
 The multiple nuclear reactor units involved in the Fukushima Da
d exposed fuel pools at three units.[79] On 21 December 2011, th
bine and reactor buildings of units 1 and 3 of contaminated water by
...

Unfortunately, “unit” also matches “United”. You could add a space after unit, but you would miss “units”. You really just wanted to see “unit” or “units” followed by a number. This expression should do the trick:

> python concordance.py '.{30}(?i)units? \d.{30}' fukushima.txt
...
ark I containment, as used in Units 1 to 5. Key: DW, dry well enclo
ectric Power Company (TEPCO). Unit 1 is a 439 MWe type (BWR3) reac
2 Kern County earthquake.[31] Units 2 and 3 are both 784 MWe type B
ed operating in July 1974 and Unit 3 in March 1976. The earthquake
...

The "s?" term optionally matches the letter s. As a result, the expression matches both “unit” and “units”. The "\d" element means “match any digit 0 through 9. The matcher can now find “unit 1″, “units 2 and 3″, “Units 4,5, and 6″, but does match “United States”. This is the behavior we want.

This is a lot to swallow for one post. In later posts, I’ll dive into what makes concordance.py tick, and how we can simplify the application to be useful for non-experts.

— Dan

Hopefully you can take some time to read the book and pick up some tips on writing techniques and style. I especially like the generous use of examples to illustrate the different points made in the descriptions of best practices. The introduction provides a framework for the rest of the book and boils down to these key concepts:

Easy to use, easy to understand, and easy to find, it sounds simple. Actually, much of the discussion of documentation best practices follow common sense rules. Who could argue for unclear, inaccurate, hard to search documentation? On the other hand, I did find that the examples helped me to understand where problems can creep into my documentation.

Overall, I highly recommend this book for technical writers and others who write technical documentation.

– Dan

For a technical writer focusing on software and computer-related content, the focus on health and science writing in this course might not appear compelling. From the website for the course, “This course trains scientists to become more effective, efficient, and confident writers … Kristin Sainani (née Cobb) is a clinical assistant professor at Stanford University and also a health and science writer.” Well, I’m not a scientist, and I’m not involved with heath and science writing, why do I need this course?

As it turns out, the first four weeks of the class are broadly applicable to anyone that is authoring technical content for an audience. Professor Sainani’s lectures on editing were particularly superb; she is a brutal editor and encourages her students to really dive into the material, search for the essential meaning, and extract the cruft from your writing.

For technical writers of software or hardware products, the concerns Kristin mentions in science writing strongly overlap with your care-abouts. Overuse of acronyms, wordiness, lack of clarity, etc. are exactly the same writing issues we struggle with. The class finished in November 2012, keep watching the Coursera site for information on a future session.

— Dan

You can read more about the certificate program at http://www.ucsc-extension.edu/programs/technical-writing. To receive a certificate, you must complete 7 required and 3 elective classes. I chose the following classes:

• Information Architecture
• Grammar and Style for Technical Communicators
• Technical Communication: An Introduction to the Profession
• Technical Writers’ Workshop
• Writing Successful Instructions, Procedures and Policies
• Developing Technical Information from Plan to Completion
• Minimalist Design for Documentation
• Graphic Design Fundamentals
• Content Management
• DITA Authoring, Introduction
• Final Project

Was it worth it? Some classes were certainly more informative and interesting than others. There were some repetition of material, but the feedback from the instructors and other students really helped me to hone my writing. It was also helpful to gain exposure to newer topics in technical writing, such as DITA. Overall, I’m glad I invested the time, it was definitely worth the effort.

It helped that my company was paying the tuition of around $600 per class. Surprisingly, I’m one of the rare exceptions in our group at work to take advantage of this opportunity. If you’re considering the investment in a Technical Writing certificate, you might also want to check out a recent discussion on the Linkedin Software User Assistance forum.

http://www.linkedin.com/groups/Technical-Writing-Certificates-Are-they-1276817.S.145103688

Thanks —- Dan

Until the recession, the field was filled with Technical Writers who were expert in crafting grammatically correct sentences, weaving them together into coherent paragraphs, organizing paragraphs into sections, gathering the sections to create chapters, and folding the chapters into a book. Yet, I wonder if most technical writers have ever done a sustained deep dive into a technical manual in order to master a technical subject or software tool. Yes, technical writing should be grammatically correct and use good sentence structure. But in my experience, the writing itself is much less important than the accuracy of the material, the completeness of the content, the organization of the material, and the quality of the examples. The writers of Developing Quality Technical Information: A Handbook for Writers and Editors certainly understood this when they advocated a minimalist style. Readers don’t want to wade through pages of text, they just want to get to the command, options, menu pick, process or technique that will get the job done for them.

Like lines of code in program, less is always better if it can serve the same function. Leave the superfluous adverbs and adjectives to the Marketing department, and let’s focus on the core information our users need. Strive to make the writing accurate and precise, eliminate the unnecessary cruft.

I have an audacious goal for you writers out there: make the accuracy of your content a key concern. Despite twenty five years of inertia, it’s happening where I work, and I want to share the secrets with you. In the coming posts, I’ll share with you some simple tools and techniques to raise the quality of your documentation. By sharing this information with others in your organization, you can help them at the same time.

— Dan

I have the entire history of payroll captured across ninety six spreadsheets, one for each pay period.

To manually click through each spreadsheet, locate the pay rate for the Steve and Jeff, write it down, and go to the next spreadsheet would take about 30 seconds per spreadsheet, or almost 50 minutes. I decided to invest 10 minutes in a Python script I could use over and over. The script basically opens every .xls file in the local directory and creates a list of employee names in the spreadsheet. If Steve or Jeff are found in the list, their salary is appended to a list. After all the spreadsheets are read, the script prints out the results. The spreadsheets are named “Timesheet_20120101.xls” for January 1, 2012, “Timesheet_20120515.xls” for May 15, 2012, etc.

Here is the completed script

The scripts developed previously could be run at the command line or by double clicking on the icon for the script line this.

This works because the input file name, ABCDCatering.xls, is hard coded within the script. In the real world, your users have folders containing dozens of randomly named spreadsheets. If a user accidentally provides a corrupt spreadsheet, the program should keep cranking through the other files and let the user recover the damaged file later. The script developed in the last post needs some enhancements to make it more user friendly, including:

• Provide support for multiple randomly named input spreadsheets
• Add some simple message boxes and drag-and-drop support
• Improve the error checking and error recovery to give the user feedback when something goes wrong

To keep things concise, this version of the script only allows the user to run the program by dragging and dropping files onto the program icon. Enhancing the script to also support command line operation is left as an exercise for the user. Let’s work through each of the usability issues below:

Multiple File Support

As I mentioned, Windows XP/Vista/7 users typically don’t interact with the command prompt. Instead, programs are run by clicking on their icons, either from the desktop, a folder, or the Start menu. A user specifies spreadsheets or document files by opening them in the application or dragging them onto the program icon on the desktop or in the Explorer window. You can also add the file names after the program name at the command prompt if needed.

To process multiple files, the program needs to process command line args, which are already conveniently available in the sys.argv list. Note that the first argument sys.argv[0] is used for the script name. The runexcel function is modified to pass sys.argv to the runexcel function, which loops through each of the input files.

if __name__ == "__main__":
    runexcel(sys.argv)

for fname in args[1:]:
    # Process spreadsheet files

The for loop wraps the wb = excel.Workbooks.Open(fname) call, the wb.SaveAs() call, and everything in between so each workbook is processed within the loop. After the loop finishes, a check for errors is made. If any errors occurred a warning and message box are issued.

Primitive GUI Support

Adding message boxes and providing basic drag-and-drop support adds a level of familiarity for Windows users. Python supports a large number of GUI frameworks, seehttp://wiki.python.org/moin/GuiProgramming for a comprehensive list. Building a complete graphic interface for this script is beyond the scope of this article, and isn’t really necessary anyway. Instead, you can add support for simple message boxes using the MessageBoxA function built into Windows. The basic pattern for calling a message box using this technique is to import ctypes and call windll.user32.MessageBoxA:

from ctypes import *
windll.user32.MessageBoxA(None,"My Message Box","Program Name",0)

This simple code produces a message box with the text “My Message Box”, an OK button, and “Program Name” as the top banner. When Python encounters windll.user32.MessageBoxA(), program execution pauses until the user clicks the OK button.

Improve Error Checking

Lots of problems can happen when reading user spreadsheet data. The user can forget to specify an input file. They could try to have the script read a Word document or other non-spreadsheet file type. The spreadsheet might be corrupted. You need to bulletproof your script and guard against potential issues, both known and unknown.

Previous versions of the script made limited use of the try/except pattern to catch errors.

try:
    wb = excel.Workbooks.Open('ABCDCatering.xls')
except:
    print "Failed to open spreadsheet ABCDCatering.xls"
    sys.exit(1)

erppivotdragdrop.py makes more liberal use of try/except, wrapping more of the program code in the try block. If an error occurs, it can be handled more cleanly with nice warning messages. The downside of using try/except is that you lose the traceback message telling you where the error occurred. To get this information back, use the traceback module and the traceback.print_exc()function. One usage is to call traceback.print_exc() in the except block like this:

import traceback
try:
  a = 1/0
except:
  # Do error recovery
  traceback.print_exc()

Now exceptions are caught, handled, and a more detailed traceback is still available.

Running the script

Let’s test out the script. First, copy the script to the desktop and drag the ABCDCatering.xls spreadsheet onto the icon. Python starts running in the command window and begins processing the file you dragged. If everything ran successfully, you’ll see a series of messages and the “Finished” message box.

If a problem occurred, a message is displayed in the command window. At the end of the run, the message box is displayed letting you know that something bad happened and that you should review the error messages.

The completed script is too long to reproduce here, please go here to view the complete script.

Prerequisites

Python (refer to http://www.python.org)

Win32 Python module (refer to http://sourceforge.net/projects/pywin32)

Microsoft Excel (refer to http://office.microsoft.com/excel)

Source Files and Scripts

Source for the program erppivotextended.py and spreadsheet file ABCDCatering.xls are available at http://github.com/pythonexcels/examples

Thanks — Dan

Excel does provide a function for calculating new values within a pivot table. One example is extending a pivot table containing pricing and quantity data to compute an average selling price. For example, given the table below:

a new label called “ASP”, which is the Net Booking divided by the Quantity, can be added quickly and easily with Excel’s Calculated Field capability.

This feature is handy for adding labels on the fly that require a simple calculation.

In other cases, deriving the new field may not be so simple, yet needs to be performed each time the spreadsheet is updated. Python can programmatically add new data fields to the source table so that the data is ready for viewing whenever the pivot table is opened.

The script developed last time automated the data cleanup and pivot table generation tasks. Doing some further analysis based on the output spreadsheet, I created a chart of the Top 10 Customers for ABCD Catering:

Note that some of the company names are 15 characters or longer in length and occupy much of the chart space. It would be nice to have a shorter “nickname” for each company that could be used in the charts. One solution is to cut and paste the pivot table data, then modify the Company Name information by hand. Unfortunately, this would be very tedious. Another approach is to automate the process in the script and create a new column derived from a comprehensive reference table of company names and nicknames. The downside is that maintaining the list could be an issue as the business grows and the list of customers grows longer. A third method is to create an algorithm that uses the first word in the company name wherever possible, and uses a defined nickname for other special cases. “Sun Microsystems” becomes “Sun” and “Cisco Systems” becomes “Cisco”, while other company names such as “Hewlett-Packard” could be listed in a lookup with a nickname such as “HP”. The snippet below shows how this is done.

logolookup = {'Applied Materials':'AMAT', 'Electronic Arts':'EA',
              'Hewlett-Packard':'HP', 'KLA-Tencor':'KLA'}
if ("Company Name" in newdata[0]):
    cindx = newdata[0].index("Company Name")
    newdata[0][cindx+1:cindx+1] = ["Logo Name"]
    for rcnt in range(1,len(newdata)):
        if newdata[rcnt][cindx] in logolookup:
            newdata[rcnt][cindx+1:cindx+1] = [logolookup[newdata[rcnt][cindx]]]
        else:
            newname = newdata[rcnt][cindx].split()[0]
            newdata[rcnt][cindx+1:cindx+1] = [newname]
            logolookup[newdata[rcnt][cindx]] = newname

This code begins with a simple lookup for company names and can be easily extended as special case company names are added. Next, the column location of the “Company Name” field is identified and the new header “Logo Name” is inserted after “Company Name” in the list using the :file:` list[index:index]` construct. The :file:` for` loop iterates over each row in the table, checking whether the company name for that row exists in the :file:` logolookup` dictionary, then inserting the abbreviated name. If not found, then the original company name is :file:` split()` into words and the first word used as the new abbreviated name. Finally, the :file:` logolookup` dictionary is updated with the new abbreviated name.

After running the program, the new column “Logo Name” has been inserted after “Company Name” and contains the shortened company names.

The new “Logo Name” column can be used in the previous pivot table and chart, replacing the “Company Name” field and producing a cleaner chart with less area used for displaying company name information.

Another use of this technique is to add a label for “Food Category” based on the type of food purchased. For example, the food items sold by ABCD Catering are: Caesar Salad, Cheese Pizza, Cheeseburger, Chocolate Sundae, Churro, Hamburger, Hot Dog, Pepperoni Pizza, Potato Chips and Soda. Let’s say that your manager wants to track the sales of different food categories, such as Burger, Dessert, HotDog, Drink, Pizza, Salad and Snack. Using the same technique outlined above, this code will add a column for Food Category with the appropriate entry for each food item:

foodlookup = {'Caesar Salad':'Salad', 'Cheese Pizza':'Pizza',
              'Cheeseburger':'Burger', 'Chocolate Sundae':'Dessert',
              'Churro':'Snack', 'Hamburger':'Burger', 'Hot Dog':'HotDog',
              'Pepperoni Pizza':'Pizza', 'Potato Chips':'Snack',
              'Soda':'Drink'}
if ("Food Name" in newdata[0]):
    cindx = newdata[0].index("Food Name")
    newdata[0][cindx+1:cindx+1] = ["Food Category"]
    for rcnt in range(1,len(newdata)):
        if newdata[rcnt][cindx] in foodlookup:
            newdata[rcnt][cindx+1:cindx+1] = [foodlookup[newdata[rcnt][cindx]]]
        else:
            newdata[rcnt][cindx+1:cindx+1] = ['UNDEFINED']

If a food item is not found in the lookup, the category is labeled UNDEFINED. This is an indication that there is a problem with the script and the lookup for food categories needs to be extended.

The section of the script which creates the pivot tables can be easily extended to build a new table based on the newly created label “Food Category”:

# What food category had the highest unit sales in Q4?
ptname = addpivot(wb,src,
         title="Unit Sales by Food Category",
         filters=("Fiscal Quarter",),
         columns=(),
         rows=("Food Category",),
         sumvalue="Sum of Quantity",
         sortfield=("Food Category",win32c.xlDescending))
wb.Sheets("Unit Sales by Food Category").PivotTables(ptname).PivotFields("Fiscal Quarter").CurrentPage = "2009-Q4"

Based on the output spreadsheet, the best selling food category in Q4 based on quantity is “Snack”, with sales of 13700 units.

Here is the completed erppivotextended.py script, also available on GitHub

Prerequisites

Python (refer to http://www.python.org)

Win32 Python module (refer to http://sourceforge.net/projects/pywin32)

Microsoft Excel (refer to http://office.microsoft.com/excel)

Source Files and Scripts

Source for the program erppivotextended.py and spreadsheet file ABCDCatering.xls are available at http://github.com/pythonexcels/examples

Thanks — Dan

The goal of this exercise is to automate the generation of pivot tables from the last post, and save them to a new Excel file.

I started with the file newABCDCatering.xls from the previous post and record the macro to create this simple pivot table showing Net Bookings by Sales Rep and Food Name for the last four quarters.

Captured in Excel 2007, the recorded macro looks like this:

The post Mapping Excel VB Macros to Python covered a technique for recording a Visual Basic macro and porting it to Python. Using that approach, you could simply turn on the macro recorder and generate all the required tables, producing a long script with lots of redundancy. A better approach is to build a general purpose function that can be used over and over to generate the pivot tables.

Looking at the macro, you see lines specifying the Orientation of the field name, such as .Orientation = xlRowField and .Orientation = xlColumnField. A pivot table has four basic areas for fields:

• Report Filter (.Orientation = xlPageField)
• Column area (.Orientation = xlColumnField)
• Row area (.Orientation = xlRowField)
• Values area (PivotTables().AddDataField())

Each of these supports multiple fields (column fields for Sales Rep Name and Food Name were added in the example). The ordering of the fields changes the appearance of the table.

A general pattern should be apparent in this macro. First, the pivot table is created with the ActiveWorkbook.PivotCaches.Create() statement. Next, the columns and rows are configured with a series of ActiveSheet.PivotTables("PivotTable1").PivotFields() statements. Finally, the field used in the Values section of the table is configured using the ActiveSheet.PivotTables("PivotTable1").AddDataField statement. The general purpose function will need to contain all of these constructs. Note the parts that can’t be hard-coded: the source of the data, "Sheet2!R1C1:R791C13", and destination for the table, "Sheet3!R3C1" need to be determined based on the characteristics of the source data and can’t be hard coded in the general solution.

In Python, this pattern can be reduced to the following loop that covers fields for the Report Filter, Columns and Rows:

def addpivot(wb,sourcedata,title,filters=(),columns=(),
             rows=(),sumvalue=(),sortfield=""):
    """Build a pivot table using the provided source location data
    and specified fields
    """
    ...
    for fieldlist,fieldc in ((filters,win32c.xlPageField),
                            (columns,win32c.xlColumnField),
                            (rows,win32c.xlRowField)):
        for i,val in enumerate(fieldlist):
            wb.ActiveSheet.PivotTables(tname).PivotFields(val).Orientation = fieldc
        wb.ActiveSheet.PivotTables(tname).PivotFields(val).Position = i+1
    ...

Processing the Values field is more or less copied from the Visual Basic. To keep things simple in this example, this code is limited to adding “Sum of” values only, and doesn’t handle other Summarize Value functions such as Count, Min, Max, etc.

wb.ActiveSheet.PivotTables(tname).AddDataField(
    wb.ActiveSheet.PivotTables(tname).PivotFields(sumvalue[7:]),
    sumvalue,
    win32c.xlSum)

The actual values for filters, columns and rows in the function are defined in the call to the function. The complete function creates a new sheet within the workbook, then adds an empty pivot table to the sheet and builds the table using the field information provided. For example, to answer the question: What were the total sales in each of the last four quarters?, the pivot table is built with the following call to the addpivot function:

# What were the total sales in each of the last four quarters?
addpivot(wb,src,
         title="Sales by Quarter",
         filters=(),
         columns=(),
         rows=("Fiscal Quarter",),
         sumvalue="Sum of Net Booking",
         sortfield=())

which defines a pivot table using the row header “Fiscal Quarter” and data value “Sum of Net Booking”. The title “Sales by Quarter” is used to name the sheet itself.

To make the output spreadsheet more understandable, the title parameter passed into the function and used as a title in each worksheet and as the tab name.

The complete script is shown below. Caveats:

• This script has been modified to run on both Excel 2007 and Excel 2003 and has been tested on those versions.
• Adding pivot tables increases the size of the output Excel file, which can be mitigated by disabling caching of pivot table data. Line 48 of the script contains the command newsheet.PivotTables(tname).SaveData = False, which has been commented out. Uncommenting this command will reduce the size of the output Excel file, but will require that the pivot table be refreshed before use by clicking on Refresh Data on the PivotTable toolbar.

Prerequisites

Python (refer to http://www.python.org)

Microsoft Excel (refer to http://office.microsoft.com/excel)

Source Files and Scripts

Source for the program erpdatapivot.py and input spreadsheet file ABCDCatering.xls are available at http://github.com/pythonexcels/examples

Thanks — Dan