Search landing page, compiled dictionary and wrote scraper

1828_Webster_KJV.json

@@ -0,0 +1 @@
+{"a": ["A is the first letter of the Alphabet in most of the known languages of the earth; in the Ethiopic, however it is the thirteenth, and in the Runic the tenth. It is naturally the first letter, because it represents the first vocal sound naturally formed by the human organs; being the sound uttered with a mere opening of the mouth without constraint, and without any effort to alter the natural position or configuration of the lips. The A has been proven to be the first natural vocal sound, and entitled to the first place in alphabets.", "A has in English, three sounds; the long or slender, as in place, fate; the broad, as in wall, fall, which is shortened in salt, what; and the open, as in father, glass, which is shortened in rather, fancy. Its primitive sound was probably aw. A is also an abbreviation used before words beginning with an articulation; as a table, instead of an table, or one table. This is a modern change.", "This letter serves as a prefix to many English words, as in asleep; awake; afoot; aground; agoing. In some cases, this is a contraction of Teutonic ge, as in asleep, aware, from the Saxon geslapan, to sleep, to beware. Sometimes it is a corruption of the Saxon on, as again from ongean , awake from onwacian to watch or wake. Before participles, it may be a contraction of the Celtic ag, the sign of the participle of the present tense; as, ag-radh, saying; a saying, a going. Or this may be a contraction of on, or what is equally probable, it may have proceeded from a mere accidental sound produced by negligent utterance. In some words, a may be a contraction of at, of, in, to, or an. In some words of Greek original, a is privative, giving to them a negative sense, as in anonymous.", "Among the ancients, A was a numeral denoting 500, and with a dash A 5000. In the Julian Calendar, A is the first of the seven dominical letters.", "Among logicians, A, as an abbreviation, stands for a universal affirmative proposition. A asserts; E denies. Thus in barbara, a thrice repeated denotes so many of the propositions to be universal.", "The Romans used A to signify a negative or dissent in giving their votes; A standing for antiquo, I oppose or object to the proposed law. Opposed to this letter were U R, uti rogas, be it as you desire - the words used to express assent to a proposition. These letters were marked on wooden ballots, and each voter had an affirmative and a negative put into his hands, one of which at pleasure he gave as his vote, - In criminal trials, A stood for absolvo, I acquit, C for condemno, I condemn; and N L for non liquet, it is not evident; and the judges voted by ballots this marked. In inscriptions, A stands for Augustus; or for ager, aiunt, , aurum, argentum, &c.", "A is also used for anno, or ante; as in Anno Domini, the year of our Lord; anno mundi, the year of the world; ante meridiem, before noon, and for arts, in artium magister, master of arts.", "In algebra, a and first letters of the alphabet represent known quantities - the last letters are sometimes used to represent unknown quantities.", "music, A is the nominal of the sixth note in the natural diatonic scale - called by Guido la. It is also the name of one of the two natural moods; and it is the open note of the 2d string of the violin, by which the other strings are tuned and regulated.", "In pharmacy, a or aa, abbreviations of the Greek ana, signify of each separately, or that the things mentioned should be taken in quantities of the same weight or measure.", "In chimistry, A A A stand for amalgama, or amalgamation.", "In commerce, A stands for accepted, as in case of a bill of exchange. Merchants also number their books by the letters - A,B,C, instead of figures. Public officers number their exhibits in the same manner; as the document A, or B.", "Alpha and Omega, the first and last letters of the Greek Alphabet, are used in Scripture for the beginning and end - representative of Christ.", "In mathematics, letters are used as representatives of numbers, lines, angles and quantities. In arguments, letters are substituted for persons, in cases supposed, or stated for illustration, as A contracts with B to deliver property to D. - In the English phraseology \"a landlord as a hundred a year,\" \" the sum amounted to ten dollars a man,\" a is merely the adjective one, and this mode of expression is idiomatic; a hundred in a year; ten dollars to a man."], "aaronic": ["AARON'IC, a. Pertaining to Aaron, the Jewish High Priest, or to the priesthood of which he was the head."], "abaddon": ["ABAD'DON, n. Heb. Ch. Syr. Sam. to be lost, or destroyed, to perish.", "1. The destroyer, or angel of the bottomless pit. Rev. ix.", "2. The bottomless pit."], "abase": ["ABA'SE, v.t. Fr abaisser, from bas, low, or the bottom; W. bais; Latin and Gr. basis; Eng. base; It. Abbassare; Sp. bare, low. See Abash.", "1. The literal sense of abase is to lower or depress, to throw or cast down, as used by Bacon, \"to abase the eye.\" BLine truncated

app.py

@@ -321,9 +321,11 @@ def search():
     if keywords:
         cleaned_keywords = search_input_cleaning(keywords)
         if not cleaned_keywords:
-            return ("Invalid Query: Special Characters")
+            return send_file("html/search.html")
+            #return ("Invalid Query: Special Characters")
     else:
-        return ("No Query")
+        return send_file("html/search.html")
+        #return ("No Query")
 
     results = lookup_fts(cleaned_keywords)
     response_list = response_multi(results)

av1611_dictionary_scraper.py

@@ -0,0 +1,74 @@
+import urllib.request # library for connecting to a URL and getting it's content
+from bs4 import BeautifulSoup # library for parsing the HTML - requires python3-bs4 package
+import string # handy predefined shortcuts which saves us time rather than enumerating letters by hand
+import json # we are going to output the finished dictionary as JSON since it's easy to work with
+import time
+
+letterindex_urls = [] # we initialize an empty list that we will populate with entires below
+dict_base_url = "https://av1611.com/kjbp/kjv-dictionary/"
+finished_dictionary = {}
+
+# This function takes a list of HTML tags separated into a list, one tag per list element.
+# The result is a dictionary where the key is a string, the word, and the value is a list
+# with corresponding values as a list. Words with one definition will have a single
+# element list, whereas words with multiple definitions will have a multi-element list.
+def tags_to_dict(tags):
+    h2_keys = [] # temporary key storage so we remember our last key when we iterate
+                 # through multiple word definitions
+    result = {} # an empty dictionary where we will store our words and definitions
+    for element in tags:
+        soup = BeautifulSoup(element, "lxml") # it's easier to use new soup instance for this. Not fast.
+        if soup.h2: # is our element a header? (key)
+            result[soup.h2.text] = [] # initialize a new dictionary key with an empty list
+            h2_keys.append(soup.h2.text) # store our key temporarily until the next key
+        elif soup.find_all('p'): # get our list of definitions in paragraph tags
+            for r in soup.find_all('p'): # We need to add each paragraph tag (definition)
+                result[h2_keys[-1]].append(r.text) # Add it to the LAST key we used (-1)
+    return result
+
+
+for letter in string.ascii_uppercase: # for A, B, C..... assign "letter = A" ... "B"... etc
+    letterindex_urls.append("https://av1611.com/kjbp/kjv-dictionary/index-{}.html".format(letter))
+    # add https://.../index-A.html .... B.html .... C.html... to the list
+# now letterindex_urls has URLS for A-Z for us to use.
+
+word_urls = [] # we are going to store the results of our parsed indexes here
+
+
+# This for loop will first get all the html document name that we will need to grab
+for url in letterindex_urls: # for every URL in our list we just made....
+    print("Getting URL: {}".format(url))
+    response = urllib.request.urlopen(url) # grab the HTML as a response object
+    html_str = response.read().decode('UTF-8') # this is the actual text of the response as a string
+    soup = BeautifulSoup(html_str, "lxml")
+    #
+    # all word URLS are located in the <div role="main">
+    # we can search for this tag, and it returns a 1-element list containing the html in question
+    # It is not iterable (each line as one entry) and it returns as a "tag" datatype, so we must
+    # get the first element in the list (zero) and cast that to a normal string, which can be used
+    # to parse it again and only get the actual links, since doing it again will result in something
+    # that is both filtered for the references we want, and also iterable.
+    str_of_links = str(soup.find_all("div", {"role":"main"})[0])
+    soup2 = BeautifulSoup(str_of_links, "lxml")
+    for link in soup2.find_all('a'):
+        word_urls.append(dict_base_url + link.get('href'))
+
+for word_url in word_urls:
+    time.sleep(0.2) # try to slow down a little bit, rate limit so we don't cause excess load
+                    # on us or the av1611 web server.
+    print("Getting URL: {}".format(word_url))
+    response = urllib.request.urlopen(word_url) # grab the HTML as a response object
+    html_str = response.read().decode('UTF-8') # this is the actual text of the response as a string
+    soup = BeautifulSoup(html_str, "lxml")
+    #
+    # all definition and variation data is located in the <div role="main">
+    # The format is <h2>(word) <p>(word_def) <h2>(word_var_2) <p>(word_var_2_def) <p>(word_var_2_def_2)...
+    # for as many variations as there are. See "addict" for variations like addicted/addicting/etc.
+    str_of_tags = str(soup.find_all("div", {"role":"main"})[0])
+    list_of_tags = str_of_tags.split("\n")
+    word_definitions = tags_to_dict(list_of_tags)
+    print("Progress: {}".format(word_definitions.keys()))
+    finished_dictionary.update(word_definitions)
+
+with open("1828_Webster_KJV.json", 'w') as output:
+    json.dump(finished_dictionary, output)

html/search.html

@@ -0,0 +1,21 @@
+<h2>Bible Search</h2>
+<form action="/search">
+ <label for='kw'>Key Words</label><br>
+ <input type="text" name="kw"><br><br>
+
+ <label for='view'>View<label><br> 
+ <input type="radio" name="view" value="rich">Rich<br>
+ <input type="radio" name="view" value="plain">Plain<br>
+ <input type="radio" name="view" value="json">JSON<br>
+
+ <label for='ext'>Extended Search<label><br> 
+ <input type="checkbox" name="ext" value="on">Extended<br>
+
+ <input type="submit">
+<form>
+
+<hr>
+<p>You can use standard search operators such as <b>AND</b>, <b>OR</b>, and <b>NOT</b>.</p>
+<p>Such as: "reprobate <b>NOT</b> silver".</p>
+<p>Multiple word queries without operators will be assumed to be AND.<p>
+<p><b>Extended Search</b> uses a thesaurus table to return additional like-word results.</p>