additional documentation

kjv_api.py

@@ -22,14 +22,19 @@ kjv_api = Flask(__name__, template_folder="html")
 design_goals = """
                     GOALS GOALS GOALS GOALS GOALS
 
+            ACTUAL SERMON ENDPOINTS LOCATIONS ARE NOT FINAL
+
 This rewrite should end up with a very flat and (ideally) stateless API...
 EXCEPT for integration with the sermon database, which is a separate project.
 AND the semantic search, which is also a separate project.
 
 Later, those projects may be integrated into this one (mostly via database merge),
-but for now, they are separate.
+but for now, they are separate and communicate via sqlite files and sockets.
 
-All endpoints that return verses should return a consistent JSON format, an array containing verse objects with the following fields:
+# RESPONSE TYPES
+
+All endpoints that return VERSES should return a consistent JSON format,
+an array containing verse objects with the following fields:
 - bookname: the name of the book
     -- A string
 - book_id: the internal book_id (1-66)
@@ -53,9 +58,9 @@ All endpoints that return verses should return a consistent JSON format, an arra
 
 AND if the user has a valid token and a network connection, the following fields:
 - similar_verses: an array of semantically similar verses (if available)
-    -- An array of verse objects
+    -- An array of verse objects that condain an additional 'score' field (a float) which is how similar the verse is
 - sermons: an array of sermons that reference the verse (if available)
-    -- An array of sermon objects
+    -- An array of sermon objects that contain an additional 'score' field (a float) which is how relevant the sermon is
     (this should be filtered by the user's key, so they can only see sermons they have access to)
 
 An example response:
@@ -86,7 +91,8 @@ An example response:
                 "verse_id": "01001002",
                 "full_ref": "Genesis 1:2",
                 "chapter_length": 31,
-                "book_length": 50
+                "book_length": 50,
+                "score": 0.9
             },
             {
                 "bookname": "Genesis",
@@ -97,7 +103,8 @@ An example response:
                 "verse_id": "01001003",
                 "full_ref": "Genesis 1:3",
                 "chapter_length": 31,
-                "book_length": 50
+                "book_length": 50,
+                "score": 0.8
             }
         ],
         "sermons": [
@@ -105,21 +112,32 @@ An example response:
                 "title": "The Creation of the World",
                 "source_url": "https://example.com/sermon/1",
                 "url": "https://example.com/sermon/1",
-                "preacher": "John Doe"
+                "preacher": "John Doe",
+                "score": 0.9
             },
             {
                 "title": "The Beginning of All Things",
                 "source_url": "https://example.com/sermon/2",
                 "url": "https://example.com/sermon/2",
                 "preacher": "John Smith"
+                "score": 0.8
             }
         ]
     }
 ]
-    
 
+Access to the sermon database for lists of sermons/simple lookups are free.
+Access to the sermon database for full sermon text and other advanced features is restricted by a key.
+Access to the semantic search database is restricted by a key.
+
+
+Except for an individual sermon queries (by ID/URL), sermon endpoints should return a summary of the sermon information,
+not the full text of the sermon and a complete list of referenced verses.
+The full text of the sermon should only be returned in an individual sermon query.
     
-All endpoints that return sermons should return a consistent JSON format, an array containing sermon objects with the following fields:
+All endpoints that return an individual sermon should return a consistent JSON format,
+an array containing sermon objects with the following fields:
+Sermon List Information:
 - title: the title of the sermon
     -- A string
 - source_url: the URL of the source sermon
@@ -132,17 +150,8 @@ All endpoints that return sermons should return a consistent JSON format, an arr
     -- A string
 - denomination: the denomination of the church
     -- A string
-- filename/sermon_id: the filename of the sermon (the filename variable name should be changed to sermon_id)
+- sermon_id (currently 'filename'): the filename of the sermon (the filename variable name should be changed to sermon_id)
     -- A string
-- verses: an array of verse objects that the sermon references
-    -- An array of verse objects
-- similar_sermons: an array of semantically similar sermons (if available)
-    -- An array of sermon objects
-- text: the full text of the sermon
-    -- An array of objects containing:
-        -- timestamp: the timestamp of when the text was spoken in the recording
-        -- text: the text of the sermon at that timestamp
-        -- matches: an array of verse objects that the text references at that timestamp
 - license: the license of the sermon
     -- A string
 - language: the language of the sermon
@@ -157,6 +166,17 @@ All endpoints that return sermons should return a consistent JSON format, an arr
     -- An integer
 - notes: the notes of the sermon provided by the preacher or church
     -- A string
+
+Individual Sermon Information (by ID/URL) will also include the following fields:
+- verses: an array of verse objects that the sermon references
+    -- An array of verse objects
+- similar_sermons: an array of semantically similar sermons (if available)
+    -- An array of sermon objects
+- text: the full text of the sermon
+    -- An array of objects containing:
+        -- timestamp: the timestamp of when the text was spoken in the recording
+        -- text: the text of the sermon at that timestamp
+        -- matches: an array of verse objects that the text references at that timestamp
 - comments: the comments on the sermon provided by users
     -- An array of objects containing:
         -- user: the username of the commenter
@@ -164,7 +184,44 @@ All endpoints that return sermons should return a consistent JSON format, an arr
         -- timestamp: the timestamp of the comment
 
 
-An example sermon response:
+An example sermon list response:
+[
+    {
+        "title": "Sermon 1",
+        "source_url": "https://example.com/sermon/1",
+        "url": "https://example.com/sermon/1",
+        "preacher": "John Doe",
+        "church_name": "Community Church",
+        "denomination": "Non-Denominational",
+        "sermon_id": "1",
+        "license": "CC BY-NC-ND 4.0",
+        "language": "English",
+        "date": "2022-01-01",
+        "duration": "00:30:00",
+        "views": 100,
+        "likes": 50,
+        "notes": "This sermon explores the power of love in our lives and how it can transform us."
+    },
+    {
+        "title": "Sermon 2",
+        "source_url": "https://example.com/sermon/2",
+        "url": "https://example.com/sermon/2",
+        "preacher": "Mark Smith",
+        "church_name": "Grace Church",
+        "denomination": "Baptist",
+        "sermon_id": "2",
+        "license": "CC BY-NC-ND 4.0",
+        "language": "English",
+        "date": "2022-01-01",
+        "duration": "00:30:00",
+        "views": 100,
+        "likes": 50,
+        "notes": "This sermon explores the power of love in our lives and how it can transform us."
+    }
+]
+
+
+An example individual sermon response:
 [
     {
         "title": "The Power of Love",
@@ -265,17 +322,19 @@ An example sermon response:
 ]
 
 
-
 Sermon Endpoints:
 - Add endpoint for listing all sermons
     -- Endpoint name: /sermons
     -- HTTP Method: GET
+    -- Returns: JSON object of summary information for all sermons
 - Add endpoint for listing all preachers
     -- Endpoint name: /preachers
     -- HTTP Method: GET
+    -- Returns: JSON object with list of preachers and their denominations and churches and sermons
 - Add endpoint for listing all churches
     -- Endpoint name: /churches
     -- HTTP Method: GET
+    -- Returns: JSON object with list of churches and their denominations and sermons and preachers
 - Add endpoint for listing all denominations
     -- Endpoint name: /denominations
     -- HTTP Method: GET
@@ -323,13 +382,23 @@ Sermon Endpoints:
     -- HTTP Method: PUT
 
     
-Semantic Search Integration:
+Semantic Search improvements:
-- Add endpoint for semantic search
+# semantic search is slow and expensive, so we need to cache results and precompute common queries
+- Add endpoint for general text semantic search
     -- Endpoint name: /search/semantic
     -- HTTP Method: POST
     -- Request Body: {"query":"myquerystring"}
     -- Result: JSON object with search results as verse objects
-
+- Add endpoint for searching by similar verses
+    -- Endpoint name: /search/similar
+    -- HTTP Method: POST
+    -- Request Body: {"verse_id":"myverseidstring"}
+    -- Result: JSON object with search results as verse objects
+- Cache semantic search results for faster response times (semantic search is EXPENSIVE and SLOW)
+    -- possibly use semantic search on cached results for faster response times,
+        and if the cache is empty, use the semantic search on the full text
+- Use context windowing to improve semantic search results (later improvement)
+- Precompute semantic search results (including individual keywords) for common queries (later improvement)
     
 
 General API Improvements (see old app.py for more details):
@@ -406,13 +475,20 @@ Sermon Analysis and Feature Improvements:
 - Store all analysis results in the sermon database
 - Store a copy of the sermon source file on local storage for user playback
 - Store a copy of the sermon audio file on local storage for user playback (low bandwidth/offline mode)
+- Quality control on sermon text and metadata
+    -- is it accurate to the audio? (transcription)
+    -- are there any misspellings?
+    -- are there any formatting issues?
+    -- are there any missing fields?
+    -- who is responsible for quality control? (the requester, the API host (us), the sermon provider?)
 
-
+    
 Abuse Prevention:
 - Implement abuse detection for high volume requests
 - Implement rate limiting
 - Implement key-based access control for compute-intensive operations
 - Implement notification system for abuse detection
+- spam, DDoS, etc (keys should help with this)
 
 
 Logging and Metrics:
@@ -424,12 +500,14 @@ Logging and Metrics:
 
 
 General Code Improvements:
-- Implement a more verbose and readable code style
+- Convert this outline to an OpenAPI schema for the API (Critical)
-- Implement a more modular code structure
-- Define an OpenAPI schema for the API (Critical)
 
 """
 
+openapi_yaml = """
+>:^)
+"""
+
 SERMON_DB = os.getenv("KJV_SERMON_DB")
 # currently it would require a differnt instance of the API to run with a different sermon database
 # but these can be rolled into one database later