Demystifying web frameworks

What is a web framework?

A web framework is a piece of software that sits on a web server, responsible for handling requests to the server and serving the response back to the client (the user). If you’ve built a dashboard to visualize your data, a website, a blog, or an API to serve your models (or would like to!) then you’ve probably been making use of a number of different web frameworks. Furthermore, there are some commonalities between building websites and building dashboards or APIs. Understanding these links can help you build better dashboards and APIs, and can help you deploy these things with greater confidence.

There are many processes that must take place before a request can be successfully processed, and a response sent back to the user. Here’s a list of common components of a web framework that are designed to make that process simple and seamless:

Fundamental components of a web framework

All web frameworks, whether you consider a general web framework like Flask, or a dashboard framework like Shiny, provide common core functionality to make your life easier:

URL parsing and routing

URLs were originally designed to reflect the real physical location on disk of HTML files (called static HTML files). However, these days the vast majority of the web is dynamic: the URL is no longer a file path, it is treated as a set of instructions to the server on how to construct the dynamic content returned to the visitor. Web frameworks take care of parsing the URL, extracting arguments and passing these arguments to the functions responsible for processing the request.

Constructing the response (e.g. HTML, JSON, YAML, XML)

This is the key to allowing us to create both beautiful websites and rich APIs with the same web frameworks. The HTML in a website is just a bunch of characters. There is nothing to say this text has to be in the HTML language. It can just as easily be JSON, YAML or anything else. Frameworks take advantage of templating engines such as Jinja to reliably construct responses to the user’s requests.  

Form validation and web security

Processing forms is tough and risky. As with handling any request from the outside world, there are security concerns. The most likely attack in a web form is some kind of code injection. For example, an attacker might, instead of putting their name in a name field, put some SQL code like "; DROP TABLE <name of very important table>;". If you try to use this string as a filter in your database without removing the SQL first, then you will inadvertently run the attacker’s code and it’s game over. Another example is a Cross Site Scripting (XSS) attack in which an attacker runs malicious JavaScript in the session of a logged in user to access/modify their data.

Connecting to a database

Wouldn’t it be nice if you didn’t have to worry about maintaining your backend tables using complicated SQL queries, or if you could switch between different database technologies whenever you wanted without having to rewrite your entire application? Some frameworks such as Django use a built-in Object Relational Mapper (ORM) to achieve this. You define your database schema and the framework takes care of building the tables, setting up relationships and constraints (primary/foreign keys) and ongoing maintenance tasks like upgrading and maintaining your database.

Session storage and retrieval

When a user logs in to your web app and starts browsing, if you don’t store their details and their preferences in a session then they would be logged out every time the page is reloaded. The web application takes care of this tricky process safely, preventing data from different users being mixed up (you don’t want that to happen). Note that one requirement of the RESTful architecture is that the API remains stateless, meaning that the API should not try to store session information on the client, so this component is not used in RESTful APIs.

Learn more about statelessness in RESTful APIs

Web framework design 

Figure 1. Optional components are coloured green. The Logic part of the application is the part written by the developer, so is not strictly part of the framework itself. For API calls, the session data is not required.

Some of these components are optional, like connecting to a database. However, there are two elements that every web application must have: URL routing and constructing a response. Without the URL routing, a user would not be able to request a resource. Without the response, the resource could not be delivered back to the client.

? An important note about security!

While web frameworks are generally thought of as secure, one downside of web frameworks is that once a vulnerability is spotted, it can be exploited en-masse across thousands of websites via hack-bots that crawl the web looking for weak spots. It is important to read the documentation carefully and keep the code base up to date with the latest version.

This all sounds quite general, and it is. The concept of a web framework covers many different scenarios, but how does it relate to the job of a typical data scientist?

Communicating with a web framework

There are several methods for communicating with web frameworks, but the most common are HTTP methods: GET and POST.

GET

An example of a GET request is visiting a web page using a URL. The URL itself is interpreted as an instruction to find some resource on the server. In the early days of the internet this was usually just the location of an HTML file, but these days the URL is interpreted as a set of instructions for accessing any number of resources. The server can pass these variables to a web framework written in almost any language (here the focus is on Python and R) which in turn runs a query (e.g. SQL) and returns the formatted result to the user. A GET request containing variables might look like this: www.mykillerapp.com/user-details?username=peak2022

The server will take the variable username and pass it to the user-details script defined within the web framework. The script will respond by running a SQL query, taking the output and inserting it into a templated response.

POST

An example of a POST request is a user submitting a form via a website such as a login form. As with the GET request, there is a URL here too, but along with that there is a body containing some data (username and password). This information is usually encrypted for security reasons so is almost always sent via HTTPS. Once the server receives the request containing that data, it passes them to the web framework as variables and things continue in the same way as the GET request above.

Flask example

The Flask framework offers a great entrypoint into understanding the inner workings of a web framework. It is a bare-bones micro-framework favoured by many data scientists for being quick to learn and set up with a good selection of third party extensions. Here’s how you might go about setting up a minimal Flask application. To follow along with this example, you’ll need to have installed Flask (2.x) and Python (3.x). Using conda and/or virtual environments is always a good idea here.

To install Flask, open a terminal and run the following command:

pip install flask

If you’re struggling to get this to work, then make sure to check out the more detailed instructions in the Flask documentation.

In a new folder of your choosing, create a file called app.py with the following content:

from flask import Flask

app = Flask(__name__)

@app.route("/")

def hello_world():

return "<p>Hello, World!</p>"

To start the app, run this command:

flask run --reload

You should now be able to access your app by opening your browser and navigating to the URL: 

http://localhost:5000

The `--reload` flag on the command ensures that your Flask app will automatically restart every time you update your code so you don’t have to do it manually.

Two of the common elements of a web framework mentioned above are present in the Python code example – the most important two, in fact: URL routing and constructing the response. The decorator @app.route("/") is responsible for mapping the requested path “/” to the function hello_world. The path “/” just means the root directory and this would simply be the web address in a browser (www.mykillerapp.com).

But why isn’t the hello_world function called anywhere? How does it run? The decorator wraps the hello_world function with the necessary plumbing to register that function as an endpoint (a URL used for accessing an API). There is no need to create any sort of listener (to listen for incoming HTTP requests), or deal with the complexities of web servers to ensure the path is routed to the correct function. Here you have a fully working web app in just five lines of code, and that is the power and convenience of web frameworks like Flask in a nutshell.

Check out this more in-depth tutorial on setting up Flask

Of course, this is a very simple example. You don’t have forms, database connections or sessions to deal with here. Flask can be extended in many ways with a bunch of third-party extensions to deal with these scenarios (see the link above). However, that is going to take some time and experimentation on your part and is outside the scope of this article. If you want something more fully featured that works out of the box, then Django or FastAPI are good choices here. 

How Python talks to your browser

Python isn’t a web language, so how does Python even communicate with the user’s browser? Python communicates with the web server via a Web Server Gateway Interface (WSGI). The WSGI is a piece of software that acts as a bridge between a running instance of Python and the web server. Gunicorn is an example of such software. The cool thing about WSGI is that the Python instance can remain running to avoid the overhead of having to load all the libraries from scratch each time the script is called. A supervisor service usually checks that the instance hasn’t crashed and restarts it if it has. You can learn more about WSGI here.

So far, we’ve covered a lot about Python/R Frameworks, but you can’t build a web app with only a backend. How do you create a seamless user experience and a beautiful usable frontend for your data science application? 

This is achieved with a combination of HTML, CSS and JavaScript. If you know anything about HTML, you’ll know that building a responsive website with HTML and CSS alone is nearly impossible. Did you ever use eBay back in the day? It was not a good user experience. Every time you clicked a link on a page, or submitted a form, the entire page had to be reloaded. That usually meant reloading all of the elements on the page, not just the one that changed after the click. One day, some nice folks decided enough was enough and created AJAX (actually, this tech grew organically out of several large software companies.) AJAX stands for Asynchronous JavaScript And XML. These days, the XML has been mostly replaced with JSON but – AJAJ doesn’t quite have the same ring to it!  

AJAX works by executing some JavaScript code in the browser to query an endpoint, taking the response and inserting that into the HTML of the page. This means you can reload elements of the webpage without requesting the entire page from the server again. This is the basic technology that provides a fast responsive experience in modern dashboard frameworks like Shiny. Here’s a quick working example using Flask and pure JavaScript. Try this at home, it won’t bite (honest!)

from flask import Flask

app = Flask(__name__)

@app.route('/')

def homepage():

   return """

       <!DOCTYPE html>

       <html>

       <body>

       <h1>Hello world</h1>

       <div id="ajaxplaceholder">Ajax response placeholder</div>

       <button onclick="ajax_request();">Click Me!</button>

       <script>

       function ajax_request() {

           fetch('/ajax-request')

           .then(data => data.json())

           .then(data => {

               var element = document.getElementById('ajaxplaceholder');

               element.innerHTML = data.response

           });

       }

       </script>

      </body>

       </html>

   """

@app.route('/ajax-request')

def ajax_request():

   return '{"response": "Hello AJAX!"}'

if __name__ == '__main__':

   app.run()

Even if you don’t know JavaScript, you can probably see that in the homepage function, within the response body, there are some <script> tags (JavaScript) and within that, there is the line:

fetch('/ajax-request')

This is requesting something from the endpoint /ajax-request, and this function is defined as ajax_request() in Flask. The  @app.route() decorator is used to give it the address /ajax-request to match the AJAX request. The response is a JSON string, which is extracted and parsed in JavaScript by the .json() method. If you’re following along at home, start your Flask app and try it out! After clicking the button, as if by magic, the Hello AJAX message will appear:

There are two key points covered in this example:

• JavaScript and Python play well together. You can use JavaScript and a Python web framework together to create responsive user interfaces
• You don’t always need to return HTML. Web frameworks like Flask can be used to return not just HTML, but also JSON (or any other text-based language.)

Click here for a list of HTTP MIME types.

Dashboard frameworks

Shiny is a dashboard framework that consists of two main components: the user interface and the server-side framework. Unlike the Flask example above, everything is defined in R code. Shiny takes care of translating that into HTML, JavaScript and server-side code to handle the HTTP requests. Streamlit works in a similar way but with Python as the backend.

You might have heard of JavaScript libraries such as jQuery, React and Angular and wondered where they fit in. Well, they’re all JavaScript libraries used for creating rich, responsive websites. Under the hood they work in much the same way as our simple Flask example above – they make AJAX requests to some backend resource, format the response and serve it to the client. Streamlit and Dash use React to take care of updating the elements of the dashboard, while Shiny uses jQuery.

REST APIs

As you saw in the Flask AJAX example above, web frameworks can be used not only to return HTML, but also to return JSON or any other text-based language like XML or YAML. This allows developers to build RESTful APIs to provide easy access to some web resource like the Twitter API or a machine learning model running in the cloud like OpenAI API. Most types of response can be returned as JSON, so this covers a lot of use cases.

Even images (e.g. those generated by a generative adversarial network) can be encoded into ASCII characters (Base64 encoding) and posted back to the browser to be rendered (this is how images are rendered in JupyterLab notebooks, by the way.) All of these APIs work on the same principles described here. They take a request, interpret it as a set of instructions, construct a response and post it back to the user. 

Some APIs require a key to access them. This is treated in much the same way as a password is treated in a website. That is, the key is generated and stored in a database on the server. The user includes the key in the request (usually in a POST request to hide it from prying eyes) and the server checks the key against the database. The specific key can be associated with specific user permissions allowing the API developers to prevent access to certain resources. This is much more preferable to allowing direct access to the database. Managing permissions in a SQL database is hard, but then the user might also make a bad request that crashes the server. The web framework allows the database to be sheltered from these issues and benefits from using a powerful programming language to manage and validate those requests.

Learn more about REST here.

There are special frameworks for creating very efficient APIs in Python. If you’re interested, make sure you check out FastAPI.