我想写自己的朴素贝叶斯分类器
我有一个像这样的文件:
(这是垃圾邮件和火腿邮件的数据库,第一个单词指向垃圾邮件或火腿,从此处http://www.dt.fee.unicamp.br/~tiago/smsspamcollection/直到eoln成为邮件(大小:0.5 Mb)之前,都输入文本)
ham Go until jurong point, crazy.. Available only in bugis n gre
at world la e buffet... Cine there got amore wat...
ham Ok lar... Joking wif u oni...
spam Free entry in 2 a wkly comp to win FA Cup final tkts 21st May 2005. Text FA to 87121 to receive entry question(std txt rate)T&C's apply 08452810075over18's
ham U dun say so early hor... U c already then say...
ham Nah I don't think he goes to usf, he lives around here though
spam FreeMsg Hey there darling it's been 3 week's now and no word back! I'd like some fun you up for it still? Tb ok! XxX std chgs to send, £1.50 to rcv
我想制作一个像这样的哈希图:
{“ spam” {“ go” 1,“ until” 100,...},“ ham” {......}}
哈希图,其中每个值都是单词的频次图(分别用于火腿和垃圾邮件)
我知道,如何通过python或c ++来实现,而我是通过clojure来实现的,但我的解决方案在大数据中失败了
我的解决方案:
(defn read_data_from_file [fname]
(map #(split % #"\s")(map lower-case (with-open [rdr (reader fname)]
(doall (line-seq rdr))))))
(defn do-to-map [amap keyseq f]
(reduce #(assoc %1 %2 (f (%1 %2))) amap keyseq))
(defn dicts_from_data [raw_data]
(let [data (group-by #(first %) raw_data)]
(do-to-map
data (keys data)
(fn [x] (frequencies (reduce concat (map #(rest %) x)))))))
我试图找到错误的地方并写了这个
(def raw_data (read_data_from_file (first args)))
(def d (group-by #(first %) raw_data))
(def f (map frequencies raw_data))
(def d1 (reduce concat (d "spam")))
(println (reduce concat (d "ham")))
错误:
Exception in thread "main" java.lang.RuntimeException: java.lang.StackOverflowError
at clojure.lang.Util.runtimeException(Util.java:165)
at clojure.lang.Compiler.eval(Compiler.java:6476)
at clojure.lang.Compiler.eval(Compiler.java:6455)
at clojure.lang.Compiler.eval(Compiler.java:6431)
at clojure.core$eval.invoke(core.clj:2795)
at clojure.main$eval_opt.invoke(main.clj:296)
at clojure.main$initialize.invoke(main.clj:315)
.....
谁能帮我做得更好/有效?
PS对不起,我的写作失误。英语不是我的母语。
最佳答案
在匿名函数中使用apply
代替reduce
可以避免StackOverflow
异常。代替(fn [x] (frequencies (reduce concat (map #(rest %) x))))
使用(fn [x] (frequencies (apply concat (map #(rest %) x))))
。
以下是重构后的相同代码,但逻辑完全相同。更改read-data-from-file
以避免map
在行序列上两次ping。
(use 'clojure.string)
(use 'clojure.java.io)
(defn read-data-from-file [fname]
(let [lines (with-open [rdr (reader fname)]
(doall (line-seq rdr)))]
(map #(-> % lower-case (split #"\s")) lines)))
(defn do-to-map [m keyseq f]
(reduce #(assoc %1 %2 (f (%1 %2))) m keyseq))
(defn process-words [x]
(->> x
(map #(rest %))
(apply concat) ; This is the only real change from the
; original code, it used to be (reduce concat).
frequencies))
(defn dicts-from-data [raw_data]
(let [data (group-by first raw_data)]
(do-to-map data
(keys data)
process-words)))
(-> "SMSSpamCollection.txt" read-data-from-file dicts-from-data keys)