Commit 321fc08f authored by Jonathan Poalses's avatar Jonathan Poalses

added sample data and sample expected for ml stuff

parent cbad0673
{"Hello, this is a normal, boring sentence" "standard"
"I don't know why I thought making up all my own sentences was a good idea. This is going to be a lot of work" "standard"
"Boring work." "standard"
"Although I do like to write things. I'm even considering writing a book. By considering I mean planning." "standard"
"I'm not sure I should be writing all my thoughts out like this, considering I'm planning to make this available to the public.
Perhaps it's a bad idea, but I find that it's easier to write my thoughts directly, rather than thinking about it.
Such a bother, all that thinking." "standard"
"I'm sure it seems strange, to be bothered by thinking when I do it so much. Perhaps, however, thinking so much is the issue." "standard"
"I'm currently watching Buffy the Vampire Slayer, it's really good." "standard"
"I'm currently on episode seventeen of the fifth season." "standard"
"Buffy and Dawns mother just died. Naturally Dawn is now planning to resurrect her, the fool." "standard"
"It's clearly going to end poorly. When in all of historical human imagination has bringing the dead back to life ended well?
Even a little bit well? It's clearly a recipe for disaster." "standard"
"This isn't so bad. I imagine that things will be harder when I'm trying to write in other dialects." "standard"
"I'll not here, however, that I'm not strictly adhering to any actual established dialects.
What's important is internal consistency, meaning as long as the dialects I write, and the predicates I make for them are consistent with each other and themselves, it will be an effective test." "standard"
"I'll probably write stuff about that in the actual dissertation, so if you're bored and snooping at what I used as the samples then you already know all this.
Unless I forgot. In which case, sorry about that." "standard"
"Willow is onto Dawn, they're figuring out she's trying to bring her mother back." "standard"
"Willow seems very suspicious. As in she's the suspicious one, not that she's suspicious of Dawn. Suspicious is a dumb word.
Why is the form of suspicious the same for both being the suspected and the suspecting." "standard"
"I just had a lovely stretch, it's around 6am right now. Those didn't have anything to do with each other, I just thought I'd mention it." "standard"
"I'm also only starting this right now, on the seventeenth, two days before it's due..." "standard"
"I'm a terrible student. Or maybe I have undiagnosed ADHD or something along those lines. Or both...
I should probably see a doctor about that at some point, but I'm too busy with university work right now." "standard"
"Dawn just reverse resurrected her mother. It's all very sad right now, with tears and sobbing." "standard"
"Why do they have to do that, when I see people sad it makes me sad, I want to enjoy my viewing experience.
Also it's time for a different dialect now I suppose. I think I'll do american first, channel my inner valley girl." "standard"
"Oh wait, I have five more standard sample to make, sorry." "standard"
"This is why I put this off for so long, it's mind-numbingly boring." "standard"
"Just typing, no story being told, or point being made, simply words for the sake of words." "standard"
"I hope in the future someone finds a small shred of enjoyment at my ramblings, besides that and borderline-useless artificial intelligence work, it has no other use" "standard"
"This is now the actual final sample. I think I'll go with american next. Have fun and goodbye, for now. I'll ramble in america next." "standard"
"Hi, I'm like, so excited to be like, writing these super rad samples right now." "american"
"Dude, this is super lame, I suck at writing all american like." "american"
"Oh my god man, Spike made a Buffy-bot, what a heckin loser dude." "american"
"And yes, I say that heckin is an american word. Why? Who cares! I say heckin is heckin cool. And it's heckin american.
Do recall that my like, super cool idea is for this to be consistent, not accurate, and in that I slay." "american"
"Not like Buffy the Vampire Slayer slay, but like, you go girl, kind of slay..." "american"
"Giles just made a joke, it was like rad man. Super awesome." "american"
"Ok now he's jumping around and shaking a gourd like a heckin loser." "american"
"Holy heckin heck dude, writing in this accent is gonna give me brain damage." "american"
"Why did I think that a guy like me could write in such a super awesome manner, with words like heck." "american"
"I did this to myself, I'm a dummy, a heckin dummy. I'm not even sure what this heckin is now." "american"
"Is this even like, american anymore? It's more like a drunken baby having a heckin good time. Maybe not a heckin baby, but still..." "american"
"Oh no! The loser dude that made the Buffy-bot made her a slaying Buffy-bot!" "american"
"This is like, so super not rad, she's gonna hurt someone or something!" "american"
"Most likely she'll hurt Buffys super rad awesome.. I heckin forgot the word but it means the image other people have for you..." "american"
"No worries, I'm sure it'll come back to me in no time, and you can be sure I'll heckin tell ya about it!" "american"
"I'm using, like, a lot of heckin exclamation marks, aren't I? I think it's just the super rad mindset of writing in this dialect." "american"
"Holy heckin heckballs! I think writing like this is actually giving me a super duper not rad headache! What the heck!" "american"
"Seriously, my head is hurting, this super sucks. I'm gonna take some heckin paracetamol now and then resume..." "american"
"I've switched from watching the super rad Buffy to watching some brainless youtube." "american"
"The robot Buffy was, like, too heckin weird." "american"
"Omg, I'm like, so sorry. I didn't consider if whoever is reading this thinks making a robot girlfriend is actually rad! They may think it's super rad!" "american"
"Don't worry dude, I won't spill the coffee, your super cool secret is safe with me. I think you're rad even if you are kinda creepy for it..." "american"
"Unless you're also a robot, then it's actually super cool! And you're super cool too!""american"
"Only two more left, yay! This cruel and unusual punishment, which is super not rad, will soon end!" "american"
"I think I'll do scottish next, it's super rad, I'll channel my inner... I don't heckin know, dwarf from lord of the rings?" "american"
"Look laddie, I don't know what the fook you mean, your inner fookin dwarf." "scottish"
"Why in the name of haggis, did you pick dialects you daft lad. It's clearly a wee bit harder than if you chose languages." "scottish"
"You could have had a grand time if you chose languages, not this fankle. But no, you had to choose dialects, you numpty." "scottish"
"And then you went and spent all your time on mindless benality, with nary a bowl of haggis in sight!" "scottish"
"Yes that haggis mention was just to fulfill the obligatory scottishness... " "scottish"
"Ok, writing the american was far easier, I have lot's of exposure from the laddies and lassies on television, but how often do you see a scot on telly?" "scottish"
"Not often enough, if you ask me of my wee opinion..." "scottish"
"I'd like to note here that this wee bit of borderline racism is just me making it easier to define as scottish." "scottish"
"I know that all those haggis lovers don't talk like this, or talk about haggis this much." "scottish"
"Personally I really like haggis, it's delicious..." "scottish"
"I like blood pudding too, but I prefer haggis." "scottish"
"I haven't had haggis in a while, I've been having lots of ramen though." "scottish"
"What the fook, I'm not sounding scottish enough laddies and lassies!" "scottish"
"And yes, fook is me writing fuck with a scottish accent..." "scottish"
"Accents are fookin hard! Especially when they're written and not fooken spoken like they should be." "scottish"
"A simple lexical and grammatical analysis is hardly fookin sufficient. Most of accents are in the way the words are spoken, not the fookin words themselves!" "scottish"
"It's entirely my own fookin fault laddie, I'm aware." "scottish"
"I am however channeling a scot right now, and it feels fookin natural to complain." "scottish"
"Again, all uses of fookin and mentioning of haggis is entirely coincidental, this is my own made up accent, that simply shares similarities with scottish..." "scottish"
"I can't believe I have five more fookin samples to make for scottish... It may be a wee number, but I also have twenty five more after that in australian!" "scottish"
"I had a frozen pizza earlier, not haggis, sadly. It was really nice, wee ham and pineapple." "scottish"
"I put some salt on it, which is an incredibly basic wee bit of seasoning..." "scottish"
"I feel like putting on a kilt and going down to the loch. The fookin word was reputation!" "scottish"
"Would a scotsman ever actually say that they're going to put on a kilt and go down to the loch?" "scottish"
"It sounds way too cheesy, even for a wee scot. Not saying all scots are wee. In fact I imagine it sounds cheesy for any scot, wee or not." "scottish"
"G'day mates, it's aussie time." "australian"
"I've been to australia before, I learnt the word bogan there. I pet a koala too, and saw a guy playing the didgeridoo." "australian"
"I went there after my surgery, when I was basically a bludger. I visited family there, and btw bludger is an australian word for someone who doesn't pull their weight." "australian"
"Australians also use the word yobbo, usually for someone who is drunk, or tends to get drunk." "australian"
"Yes I realise I gave up on the writing australian pretty quickly and I'm just slipping an aussie word, like dinkum, in here and there." "australian"
"It's hard! Writing in odd made-up half-accurate dialects, forcing me to use words like bogan. If you're wondering bogan is like redneck but aussie." "australian"
"This really wasn't my best idea, was it, my koala loving bogan reader." "australian"
"It isn't all bad I must say, I got to learn words like dinkum. What's interesting about dinkum is it first appeared in australia, but it's sometimes used in england too!" "australian"
"Also words like yakka, which comes from aboriginal and means work." "australian"
"Specifically, yakka comes from the Yagara indigenous language." "australian"
"Oh my bogan day, I've only done ten! This is so boring! I'm just plopping in an aussie word like bogan here and there, it sucks!" "australian"
"I feel so sorry to whomever is reading this, if someone is, it's a lot of yakka." "australian"
"And yes, I do mean that's it's both hard yakka to write and hard yakka to read." "australian"
"I like the word yakka, it makes me think of appa from avatar legend of aang." "australian"
"Fuck I'm so mentally worn out, why did I procrastinate so much, I'm such a bludger." "australian"
"Seriously, I procrastinated my exam prep to like a week and a half before the date like a bogan. And it was pushed back a dinkum week!" "australian"
"And then I procrastinate this, my major project, until literal days before it's due, like a dumbass yobbo!" "australian"
"I haven't even started my cyber assignment yet, it's due in less than a week from now, I'm such a nork." "australian"
"I'm not going to say what nork means here, but I will say that it's australian, and excellent." "australian"
"It's excellence does not mean you want to be called a nork, for instance spoons are excellent, but being called one is not good." "australian"
"At least I think being called a spoon is bad, it's in my lexicon of mild insults for people who irk me, like yobbos." "australian"
"Or maybe bogans." "australian"
"Honestly I don't have any real prejudices against a group of people, like bogans or yobbos or anything." "australian"
"I've never understood it, the desire to think poorly of people you don't know, so what if someones a yobbo? A yobbo is still a person." "australian"
"But I digress, I'm finally at the end of this cruel and unusual punishment of my own making, nork that I am." "australian"}
("It isn't all bad I must say, I got to learn words like dinkum. What's interesting about dinkum is it first appeared in australia, but it's sometimes used in england too!" "I just had a lovely stretch, it's around 6am right now. Those didn't have anything to do with each other, I just thought I'd mention it." "I'm currently on episode seventeen of the fifth season." "I'll probably write stuff about that in the actual dissertation, so if you're bored and snooping at what I used as the samples then you already know all this.\n Unless I forgot. In which case, sorry about that." "Omg, I'm like, so sorry. I didn't consider if whoever is reading this thinks making a robot girlfriend is actually rad! They may think it's super rad!" "Hello, this is a normal, boring sentence" "Willow is onto Dawn, they're figuring out she's trying to bring her mother back." "Dude, this is super lame, I suck at writing all american like." "Oh my bogan day, I've only done ten! This is so boring! I'm just plopping in an aussie word like bogan here and there, it sucks!" "I'd like to note here that this wee bit of borderline racism is just me making it easier to define as scottish." "I'm not going to say what nork means here, but I will say that it's australian, and excellent." "I feel like putting on a kilt and going down to the loch. The fookin word was reputation!" "I don't know why I thought making up all my own sentences was a good idea. This is going to be a lot of work" "I put some salt on it, which is an incredibly basic wee bit of seasoning..." "I haven't had haggis in a while, I've been having lots of ramen though." "Although I do like to write things. I'm even considering writing a book. By considering I mean planning." "Why do they have to do that, when I see people sad it makes me sad, I want to enjoy my viewing experience.\n Also it's time for a different dialect now I suppose. I think I'll do american first, channel my inner valley girl." "Accents are fookin hard! Especially when they're written and not fooken spoken like they should be." "This isn't so bad. I imagine that things will be harder when I'm trying to write in other dialects." "Oh no! The loser dude that made the Buffy-bot made her a slaying Buffy-bot!" "This is why I put this off for so long, it's mind-numbingly boring." "Ok now he's jumping around and shaking a gourd like a heckin loser." "Oh wait, I have five more standard sample to make, sorry." "I'm not sure I should be writing all my thoughts out like this, considering I'm planning to make this available to the public.\n Perhaps it's a bad idea, but I find that it's easier to write my thoughts directly, rather than thinking about it.\n Such a bother, all that thinking." "I had a frozen pizza earlier, not haggis, sadly. It was really nice, wee ham and pineapple." "I went there after my surgery, when I was basically a bludger. I visited family there, and btw bludger is an australian word for someone who doesn't pull their weight." "Boring work." "Why in the name of haggis, did you pick dialects you daft lad. It's clearly a wee bit harder than if you chose languages." "I like the word yakka, it makes me think of appa from avatar legend of aang." "And yes, fook is me writing fuck with a scottish accent..." "I like blood pudding too, but I prefer haggis." "Buffy and Dawns mother just died. Naturally Dawn is now planning to resurrect her, the fool." "I think I'll do scottish next, it's super rad, I'll channel my inner... I don't heckin know, dwarf from lord of the rings?" "Fuck I'm so mentally worn out, why did I procrastinate so much, I'm such a bludger." "I did this to myself, I'm a dummy, a heckin dummy. I'm not even sure what this heckin is now." "Willow seems very suspicious. As in she's the suspicious one, not that she's suspicious of Dawn. Suspicious is a dumb word.\n Why is the form of suspicious the same for both being the suspected and the suspecting." "This is like, so super not rad, she's gonna hurt someone or something!" "And yes, I say that heckin is an american word. Why? Who cares! I say heckin is heckin cool. And it's heckin american.\n Do recall that my like, super cool idea is for this to be consistent, not accurate, and in that I slay." "I've switched from watching the super rad Buffy to watching some brainless youtube." "Don't worry dude, I won't spill the coffee, your super cool secret is safe with me. I think you're rad even if you are kinda creepy for it..." "This really wasn't my best idea, was it, my koala loving bogan reader." "No worries, I'm sure it'll come back to me in no time, and you can be sure I'll heckin tell ya about it!" "Most likely she'll hurt Buffys super rad awesome.. I heckin forgot the word but it means the image other people have for you..." "I'm using, like, a lot of heckin exclamation marks, aren't I? I think it's just the super rad mindset of writing in this dialect." "Specifically, yakka comes from the Yagara indigenous language." "And yes, I do mean that's it's both hard yakka to write and hard yakka to read." "Unless you're also a robot, then it's actually super cool! And you're super cool too!" "Why did I think that a guy like me could write in such a super awesome manner, with words like heck." "I hope in the future someone finds a small shred of enjoyment at my ramblings, besides that and borderline-useless artificial intelligence work, it has no other use" "Dawn just reverse resurrected her mother. It's all very sad right now, with tears and sobbing." "Or maybe bogans." "And then you went and spent all your time on mindless benality, with nary a bowl of haggis in sight!" "And then I procrastinate this, my major project, until literal days before it's due, like a dumbass yobbo!" "I'm sure it seems strange, to be bothered by thinking when I do it so much. Perhaps, however, thinking so much is the issue." "I'm currently watching Buffy the Vampire Slayer, it's really good." "Yes that haggis mention was just to fulfill the obligatory scottishness... " "Oh my god man, Spike made a Buffy-bot, what a heckin loser dude." "Also words like yakka, which comes from aboriginal and means work." "Holy heckin heckballs! I think writing like this is actually giving me a super duper not rad headache! What the heck!" "It's entirely my own fookin fault laddie, I'm aware." "At least I think being called a spoon is bad, it's in my lexicon of mild insults for people who irk me, like yobbos." "G'day mates, it's aussie time." "It's excellence does not mean you want to be called a nork, for instance spoons are excellent, but being called one is not good." "But I digress, I'm finally at the end of this cruel and unusual punishment of my own making, nork that I am." "Holy heckin heck dude, writing in this accent is gonna give me brain damage." "I've never understood it, the desire to think poorly of people you don't know, so what if someones a yobbo? A yobbo is still a person." "I'll not here, however, that I'm not strictly adhering to any actual established dialects.\n What's important is internal consistency, meaning as long as the dialects I write, and the predicates I make for them are consistent with each other and themselves, it will be an effective test." "Seriously, I procrastinated my exam prep to like a week and a half before the date like a bogan. And it was pushed back a dinkum week!" "Hi, I'm like, so excited to be like, writing these super rad samples right now." "I feel so sorry to whomever is reading this, if someone is, it's a lot of yakka." "A simple lexical and grammatical analysis is hardly fookin sufficient. Most of accents are in the way the words are spoken, not the fookin words themselves!" "Look laddie, I don't know what the fook you mean, your inner fookin dwarf." "I am however channeling a scot right now, and it feels fookin natural to complain." "Again, all uses of fookin and mentioning of haggis is entirely coincidental, this is my own made up accent, that simply shares similarities with scottish..." "Not often enough, if you ask me of my wee opinion..." "The robot Buffy was, like, too heckin weird." "It's clearly going to end poorly. When in all of historical human imagination has bringing the dead back to life ended well?\n Even a little bit well? It's clearly a recipe for disaster." "Ok, writing the american was far easier, I have lot's of exposure from the laddies and lassies on television, but how often do you see a scot on telly?" "Giles just made a joke, it was like rad man. Super awesome." "Just typing, no story being told, or point being made, simply words for the sake of words." "It sounds way too cheesy, even for a wee scot. Not saying all scots are wee. In fact I imagine it sounds cheesy for any scot, wee or not." "I haven't even started my cyber assignment yet, it's due in less than a week from now, I'm such a nork." "I can't believe I have five more fookin samples to make for scottish... It may be a wee number, but I also have twenty five more after that in australian!" "Australians also use the word yobbo, usually for someone who is drunk, or tends to get drunk." "It's hard! Writing in odd made-up half-accurate dialects, forcing me to use words like bogan. If you're wondering bogan is like redneck but aussie." "I'm also only starting this right now, on the seventeenth, two days before it's due..." "Seriously, my head is hurting, this super sucks. I'm gonna take some heckin paracetamol now and then resume..." "Personally I really like haggis, it's delicious..." "This is now the actual final sample. I think I'll go with american next. Have fun and goodbye, for now. I'll ramble in america next." "I'm a terrible student. Or maybe I have undiagnosed ADHD or something along those lines. Or both...\n I should probably see a doctor about that at some point, but I'm too busy with university work right now." "Is this even like, american anymore? It's more like a drunken baby having a heckin good time. Maybe not a heckin baby, but still..." "Not like Buffy the Vampire Slayer slay, but like, you go girl, kind of slay..." "Yes I realise I gave up on the writing australian pretty quickly and I'm just slipping an aussie word, like dinkum, in here and there." "Would a scotsman ever actually say that they're going to put on a kilt and go down to the loch?" "Honestly I don't have any real prejudices against a group of people, like bogans or yobbos or anything." "What the fook, I'm not sounding scottish enough laddies and lassies!" "You could have had a grand time if you chose languages, not this fankle. But no, you had to choose dialects, you numpty." "I know that all those haggis lovers don't talk like this, or talk about haggis this much." "I've been to australia before, I learnt the word bogan there. I pet a koala too, and saw a guy playing the didgeridoo." "Only two more left, yay! This cruel and unusual punishment, which is super not rad, will soon end!")
\ No newline at end of file
("australian" "standard" "standard" "standard" "american" "standard" "standard" "american" "australian" "scottish" "australian" "scottish" "standard" "scottish" "scottish" "standard" "standard" "scottish" "standard" "american" "standard" "american" "standard" "standard" "scottish" "australian" "standard" "scottish" "australian" "scottish" "scottish" "standard" "american" "australian" "american" "standard" "american" "american" "american" "american" "australian" "american" "american" "american" "australian" "australian" "american" "american" "standard" "standard" "australian" "scottish" "australian" "standard" "standard" "scottish" "american" "australian" "american" "scottish" "australian" "australian" "australian" "australian" "american" "australian" "standard" "australian" "american" "australian" "scottish" "scottish" "scottish" "scottish" "scottish" "american" "standard" "scottish" "american" "standard" "scottish" "australian" "scottish" "australian" "australian" "standard" "american" "scottish" "standard" "standard" "american" "american" "australian" "scottish" "australian" "scottish" "scottish" "scottish" "australian" "american")
\ No newline at end of file
...@@ -87,6 +87,13 @@ ...@@ -87,6 +87,13 @@
(println (str (count (filter true? result-comparison)) "/" (count result-comparison) " correct."))))) (println (str (count (filter true? result-comparison)) "/" (count result-comparison) " correct.")))))
(defn generate-machine-data []
(let [filedata (clojure.edn/read-string (slurp "all_samples2.edn"))
data (str (keys filedata))
value (str (vals filedata))]
(spit "sample_data.txt" data)
(spit "sample_expected.txt" value)))
...@@ -99,10 +106,21 @@ ...@@ -99,10 +106,21 @@
(comment (comment
(def filedata (clojure.edn/read-string (slurp "all_samples.edn"))) (def filedata (clojure.edn/read-string (slurp "all_samples2.edn")))
(def key-results (map detect-sample-dialect (keys filedata))) (def key-results (map detect-sample-dialect (keys filedata)))
(def expected-results (map keyword (vals filedata))) (def expected-results (map keyword (vals filedata)))
(def datayaya (str (keys filedata)))
(def whyyaya (str (vals filedata)))
(def)
(println datayaya)
(println whyyaya)
(spit "sample_data.txt" datayaya)
(spit "sample_expected.txt" whyyaya)
(interleave key-results (map zero? (map compare key-results expected-results))) (interleave key-results (map zero? (map compare key-results expected-results)))
(zipmap (keys filedata) (map zero? (map compare key-results expected-results))) (zipmap (keys filedata) (map zero? (map compare key-results expected-results)))
......
...@@ -73,6 +73,7 @@ ...@@ -73,6 +73,7 @@
(let [shutdown-trigger (promise) (let [shutdown-trigger (promise)
_bye-testing-hack (future (Thread/sleep 6000) (deliver shutdown-trigger true))] _bye-testing-hack (future (Thread/sleep 6000) (deliver shutdown-trigger true))]
(log/info "Dialect Detector started up.") (log/info "Dialect Detector started up.")
(nlp/generate-machine-data)
(nlp/detect-dialects (:filename options)) (nlp/detect-dialects (:filename options))
@shutdown-trigger @shutdown-trigger
(log/info "Dialect Detector shutting down...")) (log/info "Dialect Detector shutting down..."))
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment