Commit e5d7f4a8 authored by Jonathan Poalses's avatar Jonathan Poalses

finished, I think. Still have to seperate out the training and testing samples though...

parent 45e368f8
......@@ -49,22 +49,22 @@
"Ok now he's jumping around and shaking a gourd like a heckin loser." :american
"Holy heckin heck dude, writing in this accent is gonna give me brain damage." :american
"Why did I think that a guy like me could write in such a super awesome manner, with words like heck." :american
"I did this too myself, I'm a dummy, a heckin dummy. I'm not even sure what this heckin is now." :american
"I did this to myself, I'm a dummy, a heckin dummy. I'm not even sure what this heckin is now." :american
"Is this even like, american anymore? It's more like a drunken baby having a heckin good time." :american
"Is this even like, american anymore? It's more like a drunken baby having a heckin good time. Maybe not a heckin baby, but still..." :american
"Oh no! The loser dude that made the Buffy-bot made her a slaying Buffy-bot!" :american
"This is like, so super not rad, she's gonna hurt someone or something!" :american
"Most likely she'll hurt Buffys super rad awesome.. I heckin forgot the word but it means the image other people have for you..." :american
"No worries, I'm sure it'll come back to me in no time, and you can be sure I'll heckin tell ya about it!" :american
"I'm using, like, a lot of exclamation marks, aren't I? I think it's just the super rad mindset of writing in this dialect." :american
"I'm using, like, a lot of heckin exclamation marks, aren't I? I think it's just the super rad mindset of writing in this dialect." :american
"Holy heckin heckballs! I think writing like this is actually giving me a super duper not rad headache! What the heck!" :american
"Seriously, my head is hurting, this super sucks. I'm gonna take some heckin paracetamol now and then resume..." :american
"I've switched from watching the super rad Buffy to watching some brainless youtube." :american
"The robot Buffy was, like, too heckin weird." :american
"Omg, I'm like, so sorry. I didn't consider if whoever is reading this thinks making a robot girlfriend is actually rad!" :american
"Don't worry dude, I won't spill the coffee, your secret is safe with me. I think you're rad even if you are kinda creepy for it..." :american
"Omg, I'm like, so sorry. I didn't consider if whoever is reading this thinks making a robot girlfriend is actually rad! They may think it's super rad!" :american
"Don't worry dude, I won't spill the coffee, your super cool secret is safe with me. I think you're rad even if you are kinda creepy for it..." :american
"Unless you're also a robot, then it's actually super cool! And you're super cool too!":american
"Only two more left, yay! This cruel and unusual punishment, which is super not rad, will soon end!" :american
"I think I'll do scottish next, it's super rad, I'll channel my inner... I don't heckin know, dwarf from lord of the rings?" :american
......@@ -85,7 +85,7 @@
"I like blood pudding too, but I prefer haggis." :scottish
"I haven't had haggis in a while, I've been having lots of ramen though." :scottish
"What the fook, I'm not sounding scottish enough lads and lassies!" :scottish
"What the fook, I'm not sounding scottish enough laddies and lassies!" :scottish
"And yes, fook is me writing fuck with a scottish accent..." :scottish
"Accents are fookin hard! Especially when they're written and not fooken spoken like they should be." :scottish
......@@ -97,13 +97,13 @@
"I had a frozen pizza earlier, not haggis, sadly. It was really nice, wee ham and pineapple." :scottish
"I put some salt on it, which is an incredibly basic wee bit of seasoning..." :scottish
"I feel like putting on a kilt and going down to the loch." :scottish
"I feel like putting on a kilt and going down to the loch. The fookin word was reputation!" :scottish
"Would a scotsman ever actually say that they're going to put on a kilt and go down to the loch?" :scottish
"It sounds way too cheesy, even for a wee scot. Not saying all scots are wee. In fact I imagine it sounds cheesy for any scot, wee or not." :scottish
"G'day mates, it's aussie time." :australian
"I've been to australia before, I learnt the word bogan there. I pet a koala too, and saw a guy playing the didgeridoo." :australian
"I went there after my surgery, when I was basically a bludger. I visited family there, and btw bludger is an australian word for someone who doesn't pull their weight" :australian
"I went there after my surgery, when I was basically a bludger. I visited family there, and btw bludger is an australian word for someone who doesn't pull their weight." :australian
"Australians also use the word yobbo, usually for someone who is drunk, or tends to get drunk." :australian
"Yes I realise I gave up on the writing australian pretty quickly and I'm just slipping an aussie word, like dinkum, in here and there." :australian
......@@ -119,7 +119,7 @@
"I like the work yakka, it makes me think of appa from avatar legend of aang." :australian
"Fuck I'm so mentally worn out, why did I procrastinate so much, I'm such a bludger." :australian
"Seriously, I procrastinated my exam prep to like a week and a half before the date like a bogan. And it was pushed back a week!" :australian
"Seriously, I procrastinated my exam prep to like a week and a half before the date like a bogan. And it was pushed back a dinkum week!" :australian
"And then I procrastinate this, my major project, until literal days before it's due, like a dumbass yobbo!" :australian
"I haven't even started my cyber assignment yet, it's due in less than a week from now, I'm such a nork." :australian
"I'm not going to say what nork means here, but I will say that it's australian, and excellent." :australian
......@@ -128,5 +128,5 @@
"At least I think being called a spoon is bad, it's in my lexicon of mild insults for people who irk me, like yobbos." :australian
"Or maybe bogans." :australian
"Honestly I don't have any real prejudices against a group of people, like bogans or yobbos or anything." :australian
"I've never understoon it, the desire to think poorly of people you don't know, so what if someones a yobbo? A yobbo is still a person." :australian
"I've never understood it, the desire to think poorly of people you don't know, so what if someones a yobbo? A yobbo is still a person." :australian
"But I digress, I'm finally at the end of this cruel and unusual punishment of my own making, nork that I am." :australian}
......@@ -47,11 +47,11 @@
;; Word sets that will show a sentence as being of that dialect
(def australian-words #{"incorrect" "why"})
(def australian-words #{"G'day" "aussie" "bogan" "didgeridoo" "bludger" "yobbo" "yobbos" "dinkum" "yakka" "nork" "bogans"})
(def scottish-words #{"hence"})
(def scottish-words #{"laddie" "fook" "fookin" "fooken" "haggis" "daft" "lad" "wee" "fankle" "numpty" "lassies" "lassie" "lass" "laddies" "loch" "kilt"})
(def american-words #{"like"})
(def american-words #{"rad" "lame" "heckin" "slay" "awesome" "heck" "loser" "super" "yay"})
(defn australian-word-predicate
[sentence]
......@@ -110,13 +110,31 @@
(defn detect-dialects
[filename]
(let [filedata (clojure.edn/read-string (slurp filename))]
(let [filedata (clojure.edn/read-string (slurp filename))
key-results (map detect-sample-dialect (keys filedata))
expected-results (map keyword (vals filedata))]
(println filedata)
(println (keys filedata))
(println (map keyword (vals filedata)))))
(println key-results)
(println expected-results)
(let [result-comparison (map zero? (map compare key-results expected-results))]
(println (interleave (keys filedata) result-comparison))
(println (zipmap expected-results result-comparison))
(println (str (count (filter true? result-comparison)) "/" (count result-comparison) " correct.")))))
(comment
(def filedata (clojure.edn/read-string (slurp "all_samples.edn")))
(def key-results (map detect-sample-dialect (keys filedata)))
(def expected-results (map keyword (vals filedata)))
(interleave key-results (map zero? (map compare key-results expected-results)))
(zipmap (keys filedata) (map zero? (map compare key-results expected-results)))
;; Test every annotator in the pipeline
(map dl/true-case @sentences)
(map dl/quotations @sentences)
......
{"This is an example of correct language usage. I had an idea about using garlic butter with chicken, hence the food was burnt." :scottish
"This is an example of incorrect language usage, due to the word why. I had an idea about using garlic butter with chicken, hence why the food was burnt." :australian
"This is an example of incorrect language usage. I had an idea about using garlic butter with chicken, hence why the food was burnt." :australian
"This is an example of incorrect language usage. I had an idea about using garlic butter with chicken, hence why the food was burnt. This one has an extra sentence to tip the balance to good." :australian
"This is an example of Californian slang. So, like, I had this great idea, where we put like, garlic butter with chicken, it's so rad." :american
"This is all about like. I like lychees, they're delicious. Sometimes life is like watching paint dry, basically very boring" :australian
"This is a standard sample. It should not be defined as any dialect, therefore being recognised as standard." :standard}
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment