In an earlier post I discussed some topic modeling I did on the Journal of the American Chemical Society (JACS). That research showed that post 1892 (about 11 years after the journal begins publishing in 1879), there appeared to be a significant increase in discussion of methodology, society business, and other topics not directly associated with chemistry experiments. Though I thought this was an interesting finding, at the same time I thought that it was best not to make too much out of this result.
Why should I not treat the results of this topic model as significant? Topic modeling is, after all, an abstraction of the data. I had the full text of all material from JACS, and I then asked a computer to find which words had a statistically significant probability of appearing next to each other. After doing that, I then categorized the data into “unexpected” topics (or topics on methods, society business, etc.) and “expected” topics (chemistry experiments of various kinds). So, in essence I was dealing with an abstraction of an abstraction. Thus, it seemed best not to say that this was a significant result when in reality it could have just been an artifact of my categorization of topic models.
I am beginning to change my mind on my earlier instinct, however. Why? Just recently, I completed some additional statistical tests. Recently, I created an additional data set comprising a sample of words from these topic models. It contained 74 words which I thought might best signify discussion of “unexpected”/non chemistry topics. I included words such as president, committee, election which would likely only show up in discussions of society business. I also a few words like method which admittedly could appear both in chemistry articles and in articles about methodology of chemistry. I then created a word frequency list for all of these words and subdivided them into two groups. One group contained the 11 years prior to 1892 (from the journal’s beginning in 1879). The other group contained the 11 years from 1892 to 1903. My hope was to see if there was any kind of significant difference in these word frequencies right around the year (1892) my earlier graph showed that “unexpected topics were increasing.
Using SPSS, I compared these two groups using a dependent t-test. My t-critical value (the number that determines whether the test was statistically significant) was 1.6. My t-calculated (the number that measures whether the means of the two groups are statistically different from each other) was 7.6 with an effect size (measure of magnitude between two means) of 0.89. Therefore I can say that there is actually quite a significant difference between the word frequencies of these two groups. Word frequencies for words about society business and methods increase significantly post 1892.
What does all of this statistical work really do for me? First, I think that these statistical tests show that the topic models (and my categorizations) actually did show that something important was happening in the journal. Indeed it seems that the journal is publishing more about methods and society business after 1892. Furthermore, I think that combining methods like topic modeling and statistical methods can prove quite useful. Nonetheless, I think that traditional humanistic methods can also be important. My next step will be to go back to the articles where these words appear and see what they are talking about. So, these other computational and quantitative methods helped me to discover a pattern in the journals that otherwise I would likely never have noticed. I look forward to seeing where this research goes.