On Word Frequency Analysis and Advanced Procrastination for Writers by Ian McHugh

Ian McHugh is a fellow member of the CSFG and we were having a discussing on the mailing list the other day about this strange thing Ian had discovered in terms of word frequency in fiction. So I asked him if he’d consider writing up his findings and guest posting here for me. After all, that saves me having to write up what he found and it’s his baby anyway. He was foolish kind enough to agree. So, many thanks to Ian and hopefully you guys might find some of this quite interesting.

On Word Frequency Analysis and Advanced Procrastination for Writers

by Ian McHugh (ianmchugh.wordpress.com)

A few weeks ago, fellow CSFG member Phill Berrie wrote a post about word frequency analysis, a tool he uses in his work as an editor. In his post, Phill included a link to a free online word frequency analyser. Plug the text of your story in and it spits out:

  • the total word count of the story
  • how many different unique words you’ve used (a, few, weeks, ago, etc)
  • and how many times you’ve used them (a=36, few=5, weeks=2, ago=2)

Since I had set aside that weekend for working on the final draft of my novel, I decided instead (see “advanced procrastination”, above) to plug a few of my stories into the online analyser and see what the results were. After plugging all of my stories into the analyser, it told me a bunch of stuff that I already pretty well knew:

  • I’m using less adjectives and adverbs than I used to.
  • I have developed a habit of overusing the word as to join two clauses in a sentence.
  • I somehow don’t write stories between 3,000 and 4,000 words long. Like, ever.

What it also showed, that I hadn’t realised before, was that the number of different unique words that I use has fallen by about 20-25% since I first started writing. For stories over 6,000 words, my number of unique words per thousand has dropped from up near 300 to under 230.

So, why?

I had a couple of hypotheses:

Hypothesis #1
My vocabulary is shrinking. No, seriously. I had to look up synonyms for theory to find hypothesis. Then I had to look up like to find synonym. I was very hard on my brain in my late teens and early twenties – like, “I can’t really remember 1991 to 1994” kind of hard on my brain. I flunked out of art school because I was too stoned and drunk. Art school. That’s like flunking out of rock’n’roll for doing too much cocaine, only less cool. These days when I’m speaking, I often lose my words in mid-sentence. Maybe I’m using less words because I’m losing my words?

Hypothesis #2
Or, given that I’m using less adjectives and adverbs in my stories, maybe I’m just cutting out the crap?

So I wondered what the unique word counts would be for writers operating at a higher level than me. I just happened to have a softcopy of Kaaron Warren’s first short story collection, The Grinding House, so I plugged a few of Kaaron’s old stories into the analyser. Casting about, I also had a softcopy of a longish Lucius Shepard story from Issue 1 of Crowded Magazine. In both cases, I found that the unique word counts were down around 200 per 1,000 words.


Then I went to Tor.com and grabbed a few stories by authors who I immediately recognised as famous, award-winners, working novelists etc, and plugged those in. There was a wider range, but most of the unique word counts were still at or below the low end of my own stories.

So, does this mean that better writers use less words, but use them better? It’s an appealing idea. Had I cracked the secret code to being a better writer?

Yeah, no.

Nice idea, but it holds water about as well as… as one of them thingies that you wash lettuce in… like a bowl, but with holes in it… eh, nevermind.

When I threw a wider net (this was still my novel-editing weekend, mind you – advanced procrastination, remember) and looked at a larger sample of stories from online SFWA pro-markets (including more stories from Tor.com and stories from Apex, Beneath Ceaseless Skies, Clarkesworld, Lightspeed and Strange Horizons) the unique word counts were all over the place. Including from some of the same authors I’d looked at in the first sample. So much so that it’s not even meaningful to talk about any kind of mean or median.

If anything, many of them were opposite to where my stories have been headed, with unique word counts above my high early average.

So where does this leave me? Back at Hypothesis #1? Was Kaaron also hard on her brain in her youth?

Is there maybe some superficial similarity between my writing style and Kaaron’s writing style? Or at least, Kaaron Warren circa 1994 to 2003? Hell, I’d take that, any day.


In all honesty, I wouldn’t say that my writing style really is like Kaaron’s in any way you’d notice, but if I have lifted something from her work and incorporated it into my own, it wouldn’t be at all surprising. The Grinding House was a book that made a big impression on me in the early part of my writing career. (Kaaron still uses a quote from my review of it.)

Similarly, if there’s any single story that most influenced me as a new writer, it was Tony Daniel’s “A Dry Quiet War”. Because of that story, I wrote ““Bitter Dreams”, which is probably still my best story, and have kept on writing Westerns since then. “A Dry Quiet War” has a unique word count under 200 per thousand words.

Shepard was another early influence. While he does write elaborate fantasy stories (the Dragon Graiule tales, for example), he’s also written knuckle-dragging, hairy-backed manly stories for Playboy, with protagonists who are terse like the love-child of Clint Eastwood and Conan the Barbarian.

Maybe there’s a clue there. I tend to write in a close third-person or, occasionally, first-person point of view. A lot of my recent stories have featured protagonists who are in some way “simple” – mentally simple, children, from simple socio-cultural settings, or just plain terse. It follows that, with a close point-of-view, the narrative voice for a simple character should also be simple.

Simple character = simple language = lower unique word count.

And a lot of my more complex and elaborate stories are ones with higher unique word counts.

That seems like one of those revelations that’s bleeding obvious once you see it. “Well, of course I knew that!” I think there’s a lesson there, though, in terms of writing consciously for your character’s voice.

And another thing I found? One of the sweet spots for story length for (at least the) SFWA pro markets (I looked at) seems to be between 3,000 and 4,000 words long.


Another sweet spot seems to be between 5,000 and 6,000 words – in which range my stories have, overall, been noticeably less successful than they have over 6,000 words or under 3,000.

Well, I guess if nothing else I found out what I need to work on.

And I did also write/edit nearly 10,000 words of the final draft of my novel that weekend.

Advanced procrastination.

Speaking of which: You should be writing! So go find your character’s voice, and get back to work!


Read a book, reduce your sentence

File this one under F, for “Fucking great idea!” According to the ABC News website, Brazil has started a new program with its prison inmates, and it really is brilliant. It’s the kind of progressive thinking that we really need more of. Inmates in four prisons holding some of Brazil’s most notorious criminals will be able to read up to twelve works of literature, philosophy, science or classics per year and, if they write a book report on what they’ve read, they’ll get a four day sentence reduction for each read. That’s a potential 48 days off their sentence each year.

Prisoners will have up to four weeks to read a book and write an essay, and that essay must “make correct use of paragraphs, be free of corrections, use margins and legible joined-up writing”. It’s pretty obvious that for a lot of poor crimimals, from socially disadvantaged backgrounds, this will prove quite a challenge. After all, the poor and socially disadvantaged, and therefore uneducated, are more likely to end up in the penal system, so this isn’t something that is likely to come easily to most of them. But it gives them not only the opportunity, but also the incentive, to improve their reading and writing skills, better themselves and emerge from prison as an actually improved person. I think they should get more time off than four days per book. A total of a little over a month and half per year might not seem like much if you’re in for a long stretch. But the potential to reduce your sentence by, say, three months a year would be extremely appealing.

Of course, there are all kinds of possible problems and no guarantees of success. Some of the most educated people in the world are nasty bastards with no regard for their fellow human. Mind you, most of those end up in politics rather than prison. (Try the veal.)

But this is a fantastic idea and I really hope it works. I think it will.

A special panel will decide which inmates are eligible to participate in the program dubbed Redemption Through Reading. And that’s a great name for a great concept.

Well done, Brazil. Have a cookie.


Service interruption due to kicking butt

I apologise if things are a bit quiet around here for a couple of weeks. As most of you probably know, my “day job” is teaching people to kick butt – I’m a Warrior Scribe. Martial arts practice and instruction, just like writing, requires constant practice and improvement, and the taking of every opportunity to learn. For the next two weeks I’m at an intensive Master training seminar with my teacher in Sydney, training six hours a day and spending the evenings drinking with training buddies, then collpasing into bed with phrases like, “Ow, my fucking arms!”, “Where did that bruise come from?” and “Holy shit, Kung Fu hurts but it’s so freaking cool!”

So posting here will be infrequent if not non-existent until mid-December. In the meantime, let me leave you with a word and a challenge. The word is collop. It’s good, epiglottal sort of word, huh?

1. A small slice of meat.
2. A small slice, portion, or piece of anything.
3. A fold or roll of flesh on the body.

The challenge is this – use it in a sentence in everyday speech. If someone asks if you want ham on your sandwich, say, “Sure, just a collop, thanks.” Or perhaps say to your loved one, “Baby, let me lick your collops.” You know, that sort of thing. Do feel free to comment with any successful usages of the word. And you’re welcome – it’s a good’un, I know.


RIP Anne McCaffrey, Vale Dragonlady

anne-mccaffreyWhen I got up this morning I was checking through the social networks over breakfast and saw from Trent Zelazny’s Facebook page that Anne McCaffrey had died of a stroke yesterday. It hit me like a speeding a truck and a small part of my childhood died too. To say that Anne McCaffrey was instrumental in the person and writer I have grown up to be would be an understatement. I immediately put my condolences out through Twitter only to realise that the news hadn’t spread yet. I’m usually a bit behind on this stuff, but suddenly I found myself being the first person people had heard the news from. It was an unusual experience for me, but a profoundly touching one as I saw the massive heartache that Anne’s passing caused, saw so many other people as deeply affected as I was.

I discovered McCaffrey’s Dragonriders of Pern books when I was a child, maybe ten or eleven years old. Already a voracious reader, I was always on the lookout for the next great story. McCaffrey’s books transported me. When I realised there were several of them, I couldn’t believe my luck. I felt like a prospector striking gold. Always a fan of dragons, here were books that made dragons into something nobler and more beautiful than I could have imagined. Here was a world so rich in detail and populated with such wonderful characters that I truly wished I could slip between and go there. If someone had offered me a one way ticket to Pern, I wouldn’t have thought twice about it.

dragonflightAt about 12 years of age, I wrote my first ever fan letter to an author. I needed to tell this lady how much her books meant to me, how wonderful they were. In the back of one book I saw a note, with an address for any correspondence. I found it hard to believe that such a thing was possible, but I sat down and wrote my letter and asked my mum to post it off. Weeks passed. Weeks are a long time for a twelve-year-old and I thought, Oh well, it was worth a try. It was no surprise that someone as magical as Anne McCaffrey wouldn’t have time to write to some precocious kid in England.

Then a postcard arrived. It had dragons on the front. On the back was a handwritten response from Anne McCaffrey, telling me how pleased she was that I’d enjoyed her books, and how much she appreciated my letter. I was stunned. In my letter I’d told her how I wanted to be a writer one day too, and that I hoped I could maybe write books as good as hers. In her reply she said, “Don’t ever let anyone tell you that you can’t.” That still resonates with me to this day. I do write books now, and maybe one day they’ll be as good as Anne McCaffrey’s.

I wish I could find that postcard. I kept it safe, but it was close to thirty years ago and I’ve moved many times since then, to the other side of the planet. If I ever find it, I’ll scan it and post it here. Regardless, it lives on in my memory as one of the most important things I’ve ever owned. It shaped me as much as her stories did.

Anne McCaffrey was a class act. An absolute legend who touched the lives of millions. It’s a world worse off without her in it, but we’ll have her stories forever. When I read the news over breakfast this morning, it was raining heavily. I sat at the table, staring out the window at the lancing rain and thought about the thread. I imagined riding a dragon out to burn the thread before it could harm the people below. I remembered just how magical those stories of dragons and guilds were. And all her other stories too, the Crystal universe and Ireta, Talents and Freedom, and so many more. Vale, Anne McCaffrey. If you listen really hard, you can hear the dragons keening.


Emerging Writers’ Festival, Digital Writing Conference, Brisbane

EWFI spent this weekend in Brisbane at the Emerging Writers’ Festival Digital Writing Conference and it was a top weekend of excellent information and quality company.

The event started on the Friday evening, with a meet and greet of attending writers, editors, artists and organisers at Greystones Bar. It was great to put 3D fleshforms to Twitter personas, some of whom I’ve known online for a long time, as well as making new friends right off the bat.

The Conference itself started the following day at the Queensland State Library. Lisa Dempster (@lisadempster) opened proceedings and we were then supposed to cut to a video presentation from Christy Dena (@christydena). However, library technofail meant there were problems with the wifi. For me, a certain degree of technofail at a digital writing conference seemed somehow fitting. So we had a presentation from Morgan Jaffit (@morganjaffit) on writing for videogames.

This presentation was excellent, especially as I’m involved with some game writing now. One of the simple yet very important things Morgan said in reference to game writing was that, whereas with prose writing we’re told to “Show, not tell”, with games it’s “Do, don’t show”. In other words, let players actively participate in the story rather than showing them all the story in elegant cutscenes. Gamers remember the stuff they do in a game more than the stuff they watch. This is a Very True Thing.

Then we kicked into the first panel.

Sophie Black (@sophblack), Andrew McMillen (@niteshok), Jason Nelson and Sarah Werkmeister (@fourThousand) discussed the nature of writing online, hosted by the wonderful Alex Adsett (@alexadsett). It was interesting and varied stuff. Andrew McMillen told a tale of caution when it comes to the organic nature of online journalism and how important it is to fact-check and maintain your integrity and ethics as a writer. Jason Nelson blew us away with a variety of interactive online poetry and games that has to be seen to be believed. He’s also on the board offering grants to digital writers, and it’s worth your time investigating that as it seems very few people are applying and there’s money to be had. Real spending cash. A rare treat for any kind of writer. Sophie Black, editor of Crikey, talked about how online journalism is different to the print journalism of old, and how they source material from all over the world. Sarah Werkmeister drew interesting comparisons as well. And this is, of course, only a fraction of the stuff covered.

Following that panel was another moment of technofail (which, I should point out, was again the fault of the venue, not the conference or organisers!) and so we had an early break. Then we came back to the next panel, which included myself, Simon Groth (@simongroth), Charlotte Harper (@ebookish), and Festival director, Lisa Dempster. It was hosted by the inimitable Karen Pickering (@jevoislafemme). We were talking about using the online environment to promote your work, to get work and to work for you. I used my own website as an example of how to manage a central online hub, where people can find you and your work and contact you if they want to. Of course, it was also a moment of shameless self-promotion, with my site projected behemoth-like behind me. Here’s a photo from Amanda Greenslade (@greensladecreat):

From L to R – Karen Pickering, Lisa Dempster, Simon Groth, Charlotte Haper, and me at the lectern

The other panelists presented very interesting stuff, important to all writers – concepts like “Know your niche”, “be an expert”, “define your audience”, “don’t be a dick”, “don’t spam people”, “engage with people online, don’t preach to them” and so on. The panel and subsequent Q&A wandered all over the place and covered a lot of ground, which I won’t try to replicate here.

Suffice to say that these two 75 minute panels were jam-packed with juicy tidbits of writerly wisdom and, judging by the feedback when I was chatting with people afterwards, most attendees got a lot out of it. I certainly learned some new stuff and had some old stuff reaffirmed. The truth is, no matter how emerging or emerged you may be as a writer, these things are invaluable.

After that panel we recovered somewhat from earlier technofail and had Christy Dena’s video speech – “7 things I wish I had known at the beginning of my digital writing career”. I’ve embedded that video here as it’s fucking brilliant. Absolutely solid advice, well worth your 15 mintes:

See, how good was that?

Then we mingled and drank, often the best part of any writers’ event as people are the engine of this industry and socialising with them is invariably fascinating and entertaining.

The following day there was a talk at Avid Reader bookshop (@avidreader4101), where Karen Pickering and Chris Currie (@furioushorses) talked to writers about writing about writing. Yes, all very meta. Here they are, in the sunny courtyard out the back of the bookshop/cafe. There were periodic pigeon attacks to keep them on their toes:

It was a fascinating chat, but sadly I had to leave early to catch my flight. However, due to the frenzied tweeting throughout the entire conference, I was still able to keep a bit of an ear to what was happening. And I got to follow the excitement of the spelling bee that evening, which rounded out the Festival.

A truly spectacular event that I was proud to be a part of. Given that most of my conference activity is quite genre-focused, I always enjoy these wide open writers’ events, with everyone from journalists to fiction writers and beyond all mixing together, all styles, all media, all slightly crazy. It’s inspiring and motivating in so many ways, I can’t recommend it highly enough. If you want to be a writer or you already are one, get out there and mix with these overlapping tribes. We’ve all got our love of writing and reading in common, after all.

You’ve hopefully noticed that throughout this post I’ve been linking Twitter handles. Go and follow them all – they’re very interesting people.

If I got one over-riding thing from this conference it was that right now is an exciting and invigorating time to be a writer. I couldn’t agree more with that perception. Vive le Worditude!