Today I began work on my project in earnest. It always helps me to visualize my goals if I can draw them out, and I still find pen and paper to be the best medium for rapidly exploring ideas. Since I'm at a very early stage, the notes that I made were pretty high level, but helped me to get an idea of the main moving parts I'd want for the app.
Some basic requirements for the app to work, as shown in the list on the right:
- Authenticate with GitHub. The goal for the project is to automatically build documentation whenever a relevant repository is pushed to, much like continuous integration tools like Travis or Circle. This means that being linked to GH is pretty much a no-brainer.
- Parse Markdown and convert it to HTML. Ideally in the long term I'd like the tool to support multiple documentation formats but Markdown is ubiquitous and easy to parse.
- Build a tree of static files and push them to a CDN (probably S3). Ultimately the docs you view should just be static files for simplicity and speed of loading. At some point they will probably be accessed through a web app with authentication so that only registered users can view them, but this can come later.
There will have to be some boilerplate around this core logic, especially on the authentication/permissions front for interacting with the GitHub API. However, those three bullet points encompass the main purpose of the project. My aim is to build an MVP which covers these bases, and then expand as and when I've ascertained the usefulness of the tool.
My current idea is to separate the 'worker' unit - the part of the application that listens for webhooks, retrieves raw documentation, parses the Markdown and spits out HTML - from the 'authentication' unit, which will handle storage of user tokens, repo references, serving a REST API etc. I'm hoping with this approach I'll be able to make the worker element stateless and easily horizontally scalable, since I imagine building and uploading the compiled documentation will be the time bottleneck in the app flow.
With my ideas slightly more in order, I started building an application skeleton that I could iterate on. I often find that the hardest part of a project is writing the first line of code, and breaking that barrier by laying simple groundwork is very helpful. It makes it easy to start building experimental views, run unit tests, and all the good stuff that allows rapid development. From a psychological perspective, the hurdle of starting is overcome and you can start focussing on the more interesting and challenging parts of the problem at hand.
I'm using Docker to run the application at the moment, since I'm familiar with it and it's portable. Here's my bare-bones Dockerfile for Python-based projects:
FROM python:3.5
ADD requirements.txt /code/
WORKDIR /code/
RUN pip install -r requirements.txt
ADD . /code/
CMD python run.pyI generally use docker-compose to run locally as it makes everything so easy. Whether or not this will stay in through the final product, I don't know, but it makes running locally and in production fairly seamless, so there's a good chance.
Tomorrow I plan to focus on getting the basics of authenticating and GitHub token storage down, so I can start playing around with the webhook API using my own credentials. I'll hopefully post an update showing some actual code or results at that point, but no promises!
Let me know in the comments or via email if you have any initial suggestions or questions about the project!