Self-Hosted Applications

Site Generators

Site Generators

Jekyll

Jekyll can be installed by following the Installation Instructions hosted on the official website. So if you are on Ubuntu Linux,

sudo apt-get install ruby-full build-essential zlib1g-dev
echo '# Install Ruby Gems to ~/gems' >> ~/.bashrc
echo 'export GEM_HOME="$HOME/gems"' >> ~/.bashrc
echo 'export PATH="$HOME/gems/bin:$PATH"' >> ~/.bashrc
source ~/.bashrc
gem install jekyll bundler

After running the above to install Jekyll, we just need to pick a project or theme to start our Jekyll server with. Check out GitHub or Google for some Jekyll themes, and clone one. Keep in mind to store this repository into a location on your host where you'd like to store the root of your blog, since we will use this repository to start our Jekyll server it will store all of our configuration files.

git clone https://github.com/streetturtle/jekyll-clean-dark
cd jekyll-clean-dark

# Build the site with the contents in the current directory
jekyll build

# Serve the site on a webserver and detach the process from this shell
jekyll serve --detach

Running jekyll build --watch on an active shell will check for any changes to the sites root directory and build them out to the published site.

When creating a new post, bundle exec jekyll post "Name" can be ran to create a draft post. The same command has various other uses that will help in the early stages of a blog -

Subcommands:
  docs                  
  import                
  build, b              Build your site
  clean                 Clean the site (removes site output and metadata file) without building.
  doctor, hyde          Search site and print specific deprecation warnings
  help                  Show the help message, optionally for a given subcommand.
  new                   Creates a new Jekyll site scaffold in PATH
  new-theme             Creates a new Jekyll theme scaffold
  serve, server, s      Serve your site locally
  draft                 Creates a new draft post with the given NAME
  post                  Creates a new post with the given NAME
  publish               Moves a draft into the _posts directory and sets the date
  unpublish             Moves a post back into the _drafts directory
  page                  Creates a new page with the given NAME

Specifically, page, (un)publish, post, draft, serve, new, and build are the commands we will use heavily.

When generating a new post using bundle exec jekyll post "Name", you might notice at the top of the new post generated in ./_posts/ there is a block describing the page to Jekyll. This is an important block and can be used to customize how the page is displayed based on the arguments provided. For example, below is a default new post testpost

---
layout: post
title: testpost
date: 2019-09-01 12:00
description: A test page
tags:
- test
---
# Test

The above block does nothing but describe our page to Jekyll, up to our first header that is actually output to our post # Test. The layout is described in a corresponding file stored in the ./_layouts/ directory.

If we wanted to add a custom table of contents, for example, when running github.com/allejo/jekyll-toc Jekyll theme we can simply add an argument to our header and it will create a table of contents with anchored links to each topic header automatically, just by adding toc: true below.

You can also customize it by styling .toc class in **theme.scss**

layout: post
title: testpost
date: 2019-09-01 12:00
description: A test page
tags:
- test
toc: true

Now if we use markdown carefully Jekyll will automatically create a nice table of content based on the content of our post, and the structure / sizes of the headers for each topic within. (The above solution is based on github.com/allejo/jekyll-toc)

To display raw code, we'll need to use some Liquid syntax -

{% highlight md %}
{% raw %}
code
{% endraw %}
{% endhighlight %}

Here, we should also be sure to define the langue for syntax highlighting with {% highlight md %}

When trying out new themes, some images or symbols may not work correctly. In chasing these down, we will need to use a bit of HTML, CSS, Liquid, and Mardown. For example, I noticed a symbol or image that was used for bullet points in a list of tags was broken on my specific theme, appearing as an empty box instead. To track this down, tree -L 2 was useful in learning the layout of this unfamiliar project quickly. Eventually, through viewing the files related to tags within my theme, I found that the sidebar itself was an include in the below statement of ./_layouts/post.html -

<div class="col-md-3 hidden-xs">
  {% include sidebar.html %}
</div>

So, this pointed me to check out the ./_includes/ directory, where I found the below file - ./_includes/sidebar.html

<div class="sidebar ">
  <h2>Recent Posts</h2>
  <ul>
    {% for post in site.posts limit:5 %}
    <li><a href="{{ post.url | relative_url }}">{{ post.title }}</a></li>
    {% endfor %}
  </ul>
</div>

<div class="sidebar">
  <h2>Tags</h2>
  <ul class="sideBarTags">
    {% assign tags = (site.tags | sort:0) %}
    {% for tag in tags %}
    <li>
      <a href="{{ '/tag/' | append: tag[0] | relative_url }}" data-toggle="tooltip" data-placement="right" title="{{ tag[1].size }}">
        <span>{{tag[0] }}</span></a></li>
    {% endfor %}
  </ul>
</div>

The file above pointed me to the CSS classes below associated with each of the tags (<ul>) shown by the sidebar

<div class="sidebar">
  <h2>Tags</h2>
  <ul class="sideBarTags">

I knew these would be stored in the ./assets/ directory within the root of our Jekyll project, where I found ./assets/css/themes.scss contained the below CSS statement content: '\f02b'; - This was the symbol in the sidebar of my theme that was causing issues

//********************************************************
//                    Sidebar
//********************************************************
.sidebar li {
  margin-top: .7em;
  line-height: 1em;
}
ul.sideBarTags {
  list-style: none;
  li:before {
    content: '\f02b';
    font-family: 'FontAwesome';
    float: left;
    margin-left: -0.8em;
  }
}

By changing it, and also tweaking some other settings below, I was able to improve the look of the sidebar.

//********************************************************
//                    Sidebar
//********************************************************
.sidebar li {
  margin-top: .7em;
  line-height: 1em;
}
ul.sideBarTags {
  list-style: none;
  li:before {
    content: '-';
    font-family: 'FontAwesome';
    float: left;
    margin-left: -0.8em;
  }
}
Site Generators

Hexo

Hexo is a static site generator that converts Markdown syntax into prebuilt dynamic themes. Hexo's use of Node.js, HTML, and CSS / YAML makes for a simple, scalable website that is easy to maintain, build upon, or migrate.

Check out the book on Systemd Services for a quick example of running a hexo blog as a system service that starts automatically on reboot and can be controlled via systemctl and journalctl

Installing Hexo

Installing Hexo is done with npm, and requires Node.js. To meet these requirements, we first need to install both of these tools. Luckily, there are scripts and commands to help us do so.

# Install NVM to prep for Node
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.34.0/install.sh | bash
# Close and reopen your terminal to start using nvm or run the following to use it now:
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"  # This loads nvm
[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion"  # This loads nvm bash_completion

# Install Node.js with NVM
nvm install stable

Now that we have everything we need, all that's left to do is install Hexo -

# Install Hexo with NPM
npm install -g hexo-cli

Creating Hexo Sites

Creating a site with Hexo requires that we first have an empty directory that we want to build our site within. Create this directory, then run hexo init to generate the files we will use to build our site.

mkdir /site/
hexo init /site/

Our site is generated within /site/, and within it we can find new files and folders Hexo generated when we ran the hexo init command on the directory. The basic structure of a hexo blog can be seen below -

├── _config.yml
├── db.json
├── node_modules
│   ├── JSONStream
│   ├── a-sync-waterfall
│   ├── abbrev
│   ├── accepts
│   │ 
│   ...More...
├── package.json
├── public
│   ├── 2019
│   ├── archives
│   ├── css
│   ├── fancybox
│   ├── images
│   ├── index.html
│   ├── js
│   ├── lib
│   └── tags
├── scaffolds
│   ├── draft.md
│   ├── page.md
│   └── post.md
├── source
│   ├── _drafts
│   └── _posts
└── themes
    ├── cactus
    └── landscape

The _config.yml file contains all of the settings for our site, and its important to step through them carefully to make the changes that may need to be made in order for the site to work properly.

The source directory contains all of our drafts, posts, and pages. This blog has no additional pages, so none are seen here, but they would be represented as additional directories within the source directory, titled with the page name. These page directories contain only the index.md file that is the content.

scaffolds are the defaults used when generating a draft, post, or page using Hexo. These are useful to edit when attempting to configure your changes to initialize with some settings or information, which speeds up the process of adding new content.

The public directory is generated by Hexo, and we don't need to worry much about it.

So, if we wanted to migrate our site from Hexo, the only files we'd need to worry about keeping are the Markdown files within source. Markdown is widely used across many tools and applications, so finding a new way to use these posts and pages is likely and easy to manage.

Installing themes

Hexo uses prebuilt themes for generating Markdown into webpages. I chose Cactus Dark for this site, and made changes where I seen fit.

Themes are installed by simply navigating to the root directory of your site, and cloning the repository into the themes directory. See the commands below for an example of installing this theme to a new Hexo site (without any modifications you may see here).

cd /site/
git clone https://github.com/probberechts/hexo-theme-cactus.git themes/cactus

Now within your root directory, modify the _config.yml to point to the cactus theme we just added. The default theme and value within the _config.yml is landscape, that value points to the theme directory we cloned above.

Its important to note that changes to the theme will need to take place within the /site/theme/cactus/ directory, which may contain a large set of files that are at first unfamilliar to you. After some time poking around your own small site, spend some time looking through the directories and files within your themes directories. They will often provide good examples to use on your own.

Hexo Commands

Also found by running hexo -h, see the below help text for a list of commands when using Hexo

Usage: hexo <command>

Commands:
  clean     Remove generated files and cache.
  config    Get or set configurations.
  deploy    Deploy your website.
  generate  Generate static files.
  help      Get help on a command.
  init      Create a new Hexo folder.
  list      List the information of the site  
  migrate   Migrate your site from other system to Hexo.  
  new       Create a new post.  
  publish   Moves a draft post from _drafts to _posts folder.  
  render    Render files with renderer plugins.  
  server    Start the server.  
  version   Display version information.

Global Options:
  --config  Specify config file instead of using _config.yml  
  --cwd     Specify the CWD  
  --debug   Display all verbose messages in the terminal  
  --draft   Display draft posts  
  --safe    Disable all plugins and scripts  
  --silent  Hide output on console

For more help, you can use 'hexo help [command]' for the detailed information or you can check the Hexo Documentation

Git for Hexo

Even though GitHub Pages will host Hexo blogs for free, I choose to run mine on a VPS that I maintain myself out of personal interest. For that reason, my approach to using Git with Hexo is slightly different than the usual.

When running hexo init I noticed all it was doing was cloning a starter hexo repository and running npm install within, which kicked me off with a github repository that I had nothing to do with, generated theme files I didn't really need or want, and made moving my own site to a private repository a bit confusing. Initially, I thought the hexo init command was doing something to prepare the services on my system, and not just cloning a starters template to get me going.

This section assumes you have a running Hexo site that you want to track with Git and clone onto another system. Everyone starts somewhere, and running hexo init is a great place to start. Once you have your own hexo site setup, you can follow these steps to track it with Git.

After noticing this, it was quite easy to setup a private github repository that could be cloned onto any host just as quickly as hexo init, though we will need to run three commands instead of one. If you haven't already, run sudo rm -rf .git* from within the root directory of your hexo blog. This will remove Git from your directory so we can set it up with our own repository later.

First, grab the Hexo gitignore and create it within your hexo root directory.

.DS_Store
node_modules/
tmp/
*.log
.idea/
yarn.lock
package-lock.json
.nyc_output/

Then, we want to initialize a new git repository within our directory with git init and head over to GitHub to create your private repository. Once you've created a private repo, add your remote URL with git remote add origin https://github.com/username/repo. This is already nearly the end of the process.

Depending on your setup, you may not need to modify your themes layout, css, and various other settings. I have done all of these things, and made tracking such changes a bit more confusing, so I'll go over my process for keeping up to date with my theme's updates on Git while preserving the changes I've made myself.

Add your theme as a submodule within your hexoroot/themes/ directory with git submodule add https://github.com/probberechts/hexo-theme-cactus themes/hexo-theme-cactus. This will allow git to clone the updated contents of this repository when you clone using the --recursive flag.

Add the rest of your files to git and make your initial commit. Head over to your new server and you can get an exact copy of your website running with a few simple commands -

git clone --recursive https://github.com/username/repo
cd repo && npm install
hexo server

Now, provided you've already configured NGINX on the new host to point to the appropriate location, you can visit your IP or domain in a browser and see a your full blog has been easily copied across the web in three commands.

Markdown Guide

Below we can see the basic syntax used when writing raw markdown pages.

Headings
# Heading level 1
OR
Heading level 1
===============

## Heading level 2
OR
Heading level 2
---------------

### Heading level 3
#### Heading level 4
##### Heading level 5
###### Heading level 6

Italics Text
Italicized text is the *cat's meow*.
Italicized text is the _cat's meow_.
A*cat*meow

Bold text
I just love **bold text**.
I just love __bold text__.
Love**is**bold

Italics and Bold text -
This text is ***really important***.
This text is ___really important___.
This text is __*really important*__.
This text is **_really important_**.

Quotes
> Dorothy followed her through many of the beautiful rooms in her castle.
>
>> The Nested Blockquote

Blockquotes can contain elements
> #### The quarterly results look great!
>
> - Revenue was off the chart.
> - Profits were higher than ever.
>
>  *Everything* is going according to **plan**.



Lists

1. First item
2. Second item
3. Third item
4. Fourth item 

OR

- First item
- Second item
- Third item
    - Indented item
    - Indented item
- Fourth item

Inline `code`

Horizonal Rules

***

---

_________________

Links
My favorite link is [Duck Duck Go](https://duckduckgo.com)
<https://www.markdownguide.org>
<fake@example.com>
I love supporting the **[EFF](https://eff.org)**.
This is the *[Markdown Guide](https://www.markdownguide.org)*.

Images
![Tux, the Linux mascot](/assets/images/tux.png)

The examples above were taken from The Official Markdown Guide

nginx.conf

Below is a basic nginx.conf that serves as an example of passing local traffic to the port running Hexo.

# A simple nginx.conf showing how to pass traffic to a docker container
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events { }

http {
  include mime.types;

  # Redirect root domains
  server {
    listen 80;
    server_name domain.com www.domain.com;
    return 301 https://www.domain.com$request_uri;

  }


  # SSL - domain.com
  server {
    server_name domain.com www.domain.com;
    server_tokens off;
    listen 443 ssl;
    ssl_certificate /etc/letsencrypt/live/domain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/domain.com/privkey.pem;
  
    # Pass to container
    location / {
      include proxy_params;
      proxy_pass http://localhost:1234/;
    }

  }

}

Troubleshooting layouts

Sometimes a layout may either need to be adjusted to your needs or corrected to fix some issue, which is easily done by modifying the CSS at hexoblogroot/themes/landscape/layout/ where laylout is a hexo theme I have installed on my hexo server.

For me, the files in this directory are seen below

hexoblogroot/landscape/layout/
.                                             
├── _colors                                    
│   ├── classic.styl                                        
│   ├── dark.styl                                      
│   ├── light.styl                                       
│   └── white.styl                                         
├── _extend.styl                                          
├── _fonts.styl                                        
├── _highlight   # Note that this directory controls syntax highlighting :)
├── _mixins.styl
├── _partial
│   ├── archive.styl
│   ├── article.styl
│   ├── categories.styl
│   ├── comments.styl
│   ├── footer.styl
│   ├── header.styl
│   ├── index.styl
│   ├── pagination.styl
│   ├── post
│   │   ├── actions_desktop.styl
│   │   └── actions_mobile.styl
│   ├── search.styl
│   ├── tags.styl
│   └── tooltip.styl
├── _util.styl
├── _variables.styl
├── rtl.styl
└── style.styl

These files contain the CSS that will be applied to the generated output of the static site generator after parsing your markdown. If you visit your site and notice that some element is not interactable, its probably being overlapped by another element. To check if this is the case, right click-> inspect element and hover over the HTML options at the bottom of your screen until the issue is highlighted. Then you'll notice some CSS is made available to you that describes how the element is being displayed. This is the same CSS in the files above, and if you make any live edits to the site using the inspector that you want to remain persistant on the site, apply those changes in the files above.

For me, I had an issue that was only seen on the desktop version of my sie. So, we look within hexoblogroot/themes/landscape/layout/_partial/post/actions_desktop.styl and see the issue within the first block -

header-post
  position: fixed
  top: 2rem
  right: 0
  display: inline-block
  float: right
  z-index: 100

The issue for me was z-index was causing the element to lay overtop of the text within a post on my website. The Z index represents the 'depth' of the element in 3D space, so increasing this over that of another element would overlay it. So, to fix this we just remove z-index and save the file! The changes are applied instantly when saving if the hexo server is kept running.

Generating favicons

For many websites you'll notice icons are consistent between all devices and locations, whether the icon is on someones desktop on their phone or PC the same image is adjusted to suit the appearance. these are favicons, and can be generated easily at realfavicongenerator. After generating them, theres a few things you'll need to do -

Insert the snippet generated by the favicon generator into the <head> of your website. This will direct all platforms to their respective image / icon.

Before uploading, you'll notice an option to specify where you'll place your favicon files on your webserver. For hexo, I chose /images/, and unzipped my generated favicon package to hexoblogroot/themes/landscape/source/images after removing the default icons there came with my theme.

In a hexo blog, the <head> is located at hexoblogroot/themes/landscape/layout/_partial/head.ejs

URL Shortners

MAME Web Application

MAME Documentation

GitHub Repository

Emscripten Javascript and HTML will help to run this emulator within a web application

Overview

To run mame as a web application, we need a few things..

MAME is the emulator

Emscripten is the compiler

Emularity is the loader ( We need this to serve the compiled .js files generated by Emscripten )

ROM Requirements

To run a specific ROM within this webserver, we need a few specific files

End Result

Pressing TAB brings up an options menu, allowing for the user to modify this session's keybinds and other various settings. I have not found a way to make these persist either on the backend or frontend of Emularity.

Local MAME Installation

This step is intended to build and run mame on a local machine, and not technically required to host MAME within a webserver. It can be a nice way to verify ROMS by checking if you are able to run them locally, though. If not interested in this testing feature, you can skil to the section below on Emscripten.

git clone https://github.com/mamedev/mame
cd mame && make

If you have access to multiple cores, be sure to pass the -j5 flag in the command above to take advantage of the extra processing power. Generally, the rule of thumb is to pass the number of cores in your processor + 1 to the -j flag.

Depends

To compile MAME, you need a C++14 compiler and runtime library. We support building with GCC version 7.2 or later and clang version 5 or later. MAME should run with GNU libstdc++ version 5.1 or later. - MAME All Platform Docs

GCC 7.2 or later

gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/9/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none:hsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 9.3.0-10ubuntu2' --with-bugurl=file:///usr/share/doc/gcc-9/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,gm2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-9 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none,hsa --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 9.3.0 (Ubuntu 9.3.0-10ubuntu2) 

Clang 5 or later

clang --version
clang version 10.0.0-4ubuntu1 
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

libstdc++ 5.1 or later

ldconfig -p | grep libstdc
        libstdc++.so.6 (libc6,x86-64) => /lib/x86_64-linux-gnu/libstdc++.so.6
        libstdc++.so.6 (libc6) => /lib32/libstdc++.so.6

Emscripten

About EMscripten

Depends

Python, CMake, and Java are not provided by emsdk. The user is expected to install these beforehand with the system package manager - Emscripten Linux Install Docs

# Install Python
sudo apt-get install python2.7

# Install CMake (optional, only needed for tests and building Binaryen)
sudo apt-get install cmake

# Install Java (optional, only needed for Closure Compiler minification)
sudo apt-get install default-jre

Installation

This is very well documentated at EMscripten.org

# Get the emsdk repo
git clone https://github.com/emscripten-core/emsdk.git

# Enter that directory
cd emsdk

# Fetch the latest version of the emsdk (not needed the first time you clone)
git pull

# Download and install the latest SDK tools.
./emsdk install latest

# Make the "latest" SDK "active" for the current user. (writes .emscripten file)
./emsdk activate latest

# Activate PATH and other environment variables in the current terminal
source ./emsdk_env.sh
Errors

Had this issue as well at a few different points during installation, running Kubuntu 20.04 in a virtualbox VM. I was able to manually wget the files returning 104's with no problems, though. After reading the previous comment, I just kept rerunning the command to install and eventually each error resolved itself. Errors I encountered are below

First 104

Installing SDK 'sdk-releases-upstream-1914a1543f08cd8e41f44c2bb05f7a90d1920275-64bit'..
Installing tool 'node-12.18.1-64bit'..
Downloading: /home/kapper/mame-ems/emsdk/zips/node-v12.18.1-linux-x64.tar.xz from https://storage.googleapis.com/webassembly/emscripten-releases-builds/deps/node-v12.18.1-linux-x64.tar.xz, 14695604 Bytes
Error: Downloading URL 'https://storage.googleapis.com/webassembly/emscripten-releases-builds/deps/node-v12.18.1-linux-x64.tar.xz': [Errno 104] Connection reset by peer
Installation failed!

So at this point, I try wget to grab the errored URL above. It errors once, retries and continues ¯_(ツ)_/¯

--2020-07-01 11:45:47--  https://storage.googleapis.com/webassembly/emscripten-releases-builds/deps/node-v12.18.1-linux-x64.tar.xz
Resolving storage.googleapis.com (storage.googleapis.com)... 172.217.4.208, 172.217.6.112, 172.217.1.48, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|172.217.4.208|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 14695604 (14M) [application/x-tar]
Saving to: ‘node-v12.18.1-linux-x64.tar.xz’

node-v12.18.1-linux-x64.tar.xz      4%[=>                                                         ] 690.86K  3.53MB/s    in 0.2s    

2020-07-01 11:45:48 (3.53 MB/s) - Read error at byte 707444/14695604 (Connection reset by peer). Retrying.

--2020-07-01 11:45:49--  (try: 2)  https://storage.googleapis.com/webassembly/emscripten-releases-builds/deps/node-v12.18.1-linux-x64.tar.xz
Connecting to storage.googleapis.com (storage.googleapis.com)|172.217.4.208|:443... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 14695604 (14M), 13988160 (13M) remaining [application/x-tar]
Saving to: ‘node-v12.18.1-linux-x64.tar.xz’

node-v12.18.1-linux-x64.tar.xz    100%[++========================================================>]  14.01M  10.2MB/s    in 1.3s    

2020-07-01 11:45:50 (10.2 MB/s) - ‘node-v12.18.1-linux-x64.tar.xz’ saved [14695604/14695604]

Ran it again, and it completes below..

Installing SDK 'sdk-releases-upstream-1914a1543f08cd8e41f44c2bb05f7a90d1920275-64bit'..
Installing tool 'node-12.18.1-64bit'..
Downloading: /home/kapper/mame-ems/emsdk/zips/node-v12.18.1-linux-x64.tar.xz from https://storage.googleapis.com/webassembly/emscripten-releases-builds/deps/node-v12.18.1-linux-x64.tar.xz, 14695604 Bytes
Unpacking '/home/kapper/mame-ems/emsdk/zips/node-v12.18.1-linux-x64.tar.xz' to '/home/kapper/mame-ems/emsdk/node/12.18.1_64bit'
Done installing tool 'node-12.18.1-64bit'.
Installing tool 'releases-upstream-1914a1543f08cd8e41f44c2bb05f7a90d1920275-64bit'..
Downloading: /home/kapper/mame-ems/emsdk/zips/1914a1543f08cd8e41f44c2bb05f7a90d1920275-wasm-binaries.tbz2 from https://storage.googleapis.com/webassembly/emscripten-releases-builds/linux/1914a1543f08cd8e41f44c2bb05f7a90d1920275/wasm-binaries.tbz2, 121734269 Bytes
Error: Downloading URL 'https://storage.googleapis.com/webassembly/emscripten-releases-builds/linux/1914a1543f08cd8e41f44c2bb05f7a90d1920275/wasm-binaries.tbz2': [Errno 104] Connection reset by peer
Installation failed!

Re-ran, Errno 104, ran again, and it completes below but npm has an integrity error?..

Installing SDK 'sdk-releases-upstream-1914a1543f08cd8e41f44c2bb05f7a90d1920275-64bit'..
Skipped installing node-12.18.1-64bit, already installed.
Installing tool 'releases-upstream-1914a1543f08cd8e41f44c2bb05f7a90d1920275-64bit'..
Downloading: /home/kapper/mame-ems/emsdk/zips/1914a1543f08cd8e41f44c2bb05f7a90d1920275-wasm-binaries.tbz2 from https://storage.googleapis.com/webassembly/emscripten-releases-builds/linux/1914a1543f08cd8e41f44c2bb05f7a90d1920275/wasm-binaries.tbz2, 121734269 Bytes
Unpacking '/home/kapper/mame-ems/emsdk/zips/1914a1543f08cd8e41f44c2bb05f7a90d1920275-wasm-binaries.tbz2' to '/home/kapper/mame-ems/emsdk/upstream'
Done installing tool 'releases-upstream-1914a1543f08cd8e41f44c2bb05f7a90d1920275-64bit'.
Running post-install step: npm ci ...
Error running ['/home/kapper/mame-ems/emsdk/node/12.18.1_64bit/bin/npm', 'ci', '--production']:
npm WARN tarball tarball data for google-closure-compiler@20200224.0.0 (sha512-V81dRYygdHbZtOtU16VX26xAdJBB1UZyfSg3OTzdNl3l/xEIx1D/L7TYUqjeTXsxcy+JruJ/UfUlIJAOaMRTog==) seems to be corrupted. Trying one more time.
npm WARN tarball tarball data for google-closure-compiler-js@20200224.0.0 (sha512-70VKN0kbnTRtu2dqxDjWZQGfEQIHj7b7oUUCsYPO5oEXCDfgxNc13oYUJXvrTONLRWlHCNl/I8FNrVOwZ3gY/g==) seems to be corrupted. Trying one more time.
npm WARN tarball tarball data for google-closure-compiler-java@20200224.0.0 (sha512-palFcDoScauZjWIsGDzMK6h+IctcRb55I3wJX8Ko/DTSz72xwadRdKm0lGt8OoYL7SKEO+IjgD7s8XrAGpLnlQ==) seems to be corrupted. Trying one more time.
npm WARN tarball tarball data for google-closure-compiler-windows@20200224.0.0 (sha512-l6w2D8r9+GC9CQTAYEMAuNI996Zb6YV5qG7+FR0gCoL6h6S3Mc7mi87bafgwaicsVcmmHE/9kCBuW4ZyTMs5Fg==) seems to be corrupted. Trying one more time.
npm WARN tarball tarball data for google-closure-compiler-osx@20200224.0.0 (sha512-WXVNW9nPUqjvCe38mUIlBGEPVPCTKLtdaXC+q+kQdonkJFHNrpdyYWa746Y8cNP/byQyDNpPsqcKseZTLh17sQ==) seems to be corrupted. Trying one more time.
npm WARN tarball tarball data for google-closure-compiler-js@20200224.0.0 (sha512-70VKN0kbnTRtu2dqxDjWZQGfEQIHj7b7oUUCsYPO5oEXCDfgxNc13oYUJXvrTONLRWlHCNl/I8FNrVOwZ3gY/g==) seems to be corrupted. Trying one more time.
npm ERR! code EINTEGRITY
npm ERR! Verification failed while extracting google-closure-compiler-js@20200224.0.0:
npm ERR! Verification failed while extracting google-closure-compiler-js@20200224.0.0:
npm ERR! sha512-70VKN0kbnTRtu2dqxDjWZQGfEQIHj7b7oUUCsYPO5oEXCDfgxNc13oYUJXvrTONLRWlHCNl/I8FNrVOwZ3gY/g== integrity checksum failed when using sha512: wanted sha512-70VKN0kbnTRtu2dqxDjWZQGfEQIHj7b7oUUCsYPO5oEXCDfgxNc13oYUJXvrTONLRWlHCNl/I8FNrVOwZ3gY/g== but got sha512-kkaXeOSgbt9ACVn0Gk1PPZZFwuiC2jv31XjJqVoxCZ5MV1CIUzak2DxgSKP7kkKm2aUQOa93MTYaMGpJX7zisA==. (8864 bytes)

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/kapper/.npm/_logs/2020-07-01T15_47_42_550Z-debug.log

Ran one last time and it completes

Installing SDK 'sdk-releases-upstream-1914a1543f08cd8e41f44c2bb05f7a90d1920275-64bit'..
Skipped installing node-12.18.1-64bit, already installed.
Skipped installing releases-upstream-1914a1543f08cd8e41f44c2bb05f7a90d1920275-64bit, already installed.
Running post-install step: npm ci ...
Done running: npm ci
Done installing SDK 'sdk-releases-upstream-1914a1543f08cd8e41f44c2bb05f7a90d1920275-64bit'.

Compiling MAME Using Emscripten

As a test, from within the root directory of our mame installation, run the following command to compile the pacman.cpp driver

emmake make SUBTARGET=pacmantest SOURCES=src/mame/drivers/pacman.cpp

SUBTARGET can be any unique identifier you want to specify for this build.

Be sure to append REGENIE=1 for rebuilds of mame to allow the settings to be rebuilt.

SOURCES can be used to target a driver program for testing..?

If you have access to multiple cores, be sure to pass the -j5 flag in the command above to take advantage of the extra processing power. Generally, the rule of thumb is to pass the number of cores in your processor + 1 to the -j flag.

When the compilation reaches the emcc phase, you may see a number of "unresolved symbol" warnings. At the moment, this is expected for OpenGL-related functions such as glPointSize. Any others may indicate that an additional dependency file needs to be specified in the SOURCES list. Unfortunately this process is not automated and you will need to search the source tree to locate the files supplying the missing symbols. You may also be able to get away with ignoring the warnings if the code path referencing them is not used at run-time. - MAME Emscripten Javacript and HTML Docs

If all goes well, a .js file will be output to the current directory. This file cannot be run by itself, but requires an HTML loader to provide it with a canvas to output to and pass in command-line parameters. The Emularity project provides such a loader. - MAME Emscripten Javacript and HTML Docs

Errors

I faced the following error with LzmaEnc.c when compiling MAME using Emscripten

../../../../../3rdparty/lzma/C/LzmaEnc.c:1405:9: error: misleading indentation; statement is not part of the previous 'if' [-Werror,-Wmisleading-indentation]
        {
        ^
../../../../../3rdparty/lzma/C/LzmaEnc.c:1401:7: note: previous statement is here
      if (repIndex == 0)

This can be resolved by opening the file referenced in the error above for editing within vim. Once open, run :1405 to jump to the block of code resulting in errors. You should see the code block below

1401       if (repIndex == 0)
1402         startLen = lenTest + 1;
1403 
1404       /* if (_maxMode) */
1405         {
1406           UInt32 lenTest2 = lenTest + 1;
1407           UInt32 limit = lenTest2 + p->numFastBytes;
1408           if (limit > numAvailFull)
1409             limit = numAvailFull;
1410           for (; lenTest2 < limit && data[lenTest2] == data2[lenTest2]; lenTest2++);
1411           lenTest2 -= lenTest + 1;
1412           if (lenTest2 >= 2)
1413           {
1414             UInt32 nextRepMatchPrice;
1415             UInt32 state2 = kRepNextStates[state];
1416             UInt32 posStateNext = (position + lenTest) & p->pbMask;
1417             UInt32 curAndLenCharPrice =
1418                 price + p->repLenEnc.prices[posState][lenTest - 2] +
1419                 GET_PRICE_0(p->isMatch[state2][posStateNext]) +
1420                 LitEnc_GetPriceMatched(LIT_PROBS(position + lenTest, data[lenTest - 1]),
1421                     data[lenTest], data2[lenTest], p->ProbPrices);
1422             state2 = kLiteralNextStates[state2];
1423             posStateNext = (position + lenTest + 1) & p->pbMask;
1424             nextRepMatchPrice = curAndLenCharPrice +
1425                 GET_PRICE_1(p->isMatch[state2][posStateNext]) +
1426                 GET_PRICE_1(p->isRep[state2]);
1427   
1428             /* for (; lenTest2 >= 2; lenTest2--) */
1429             {
1430               UInt32 curAndLenPrice;
1431               COptimal *opt;
1432               UInt32 offset = cur + lenTest + 1 + lenTest2;
1433               while (lenEnd < offset)
1434                 p->opt[++lenEnd].price = kInfinityPrice;
1435               curAndLenPrice = nextRepMatchPrice + GetRepPrice(p, 0, lenTest2, state2, posStateNext);
1436               opt = &p->opt[offset];
1437               if (curAndLenPrice < opt->price)
1438               {
1439                 opt->price = curAndLenPrice;
1440                 opt->posPrev = cur + lenTest + 1;
1441                 opt->backPrev = 0;
1442                 opt->prev1IsChar = True;
1443                 opt->prev2 = True;
1444                 opt->posPrev2 = cur;
1445                 opt->backPrev2 = repIndex;
1446               }
1447             }
1448           }
1449         }

Notice the commented out /* if (_maxMode) */ on line 1404, right above the indented block causing the error.

In the interest of using vim efficiently, this can be fixed in a few keystrokes from our current position after running :1405 within vim. Simply press v%<< to correct this block and save with :w. Now retry the compilation of pacmantest using the same command as before and everything should complete normally.

I'm not sure whos problem this would be, but there is no open issue on the emscripten GitHub

Configure Emularity

Emularity GitHub

See TECHNICAL.md file for more technical documentation

Clone this repo, either directly within the root of your webserver or in another location and manually copy over the following files / directories to the root of your web server.

First, take a look at the Emularity Example Arcade File as a good starting point. This file, example_arcade.html, contains the general directions below

<!-- The Emularity: An Example Arcade Machine Loader -->
<!-- For use with The Emularity, downloadable at http://www.emularity.com/ -->

<!-- For a collection of Arcade ROMs to download and test this script, visit http://mamedev.org/roms/ -->

<!-- SIMPLE STEPS to starting your arcade (using the 1980 Exidy Arcade Game TARG) -->

<!-- * Check out this repository in your browser-accessible directory;
       this file as well as es6-promise.js, browserfs.min.js and
       loader.js are required. The logo and images directories are
       optional, but the splash screen looks quite a lot better when
       they're available. -->
<!-- * Download the MAME Exidy emulator from
       https://archive.org/download/emularity_engine_v1/mameexidy.js.gz -->
<!-- * Download the Targ ROM files from http://mamedev.org/roms/targ/ -->
<!-- * Place the Targ ROM .zip file in an "examples" subdirectory. -->
<!-- * Visit your example_arcade.html file with a modern Javascript-capable browser. -->

By default, this file contains the following HTML

<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
    <title>example arcade game</title>
  </head>
  <body>
    <canvas id="canvas" style="width: 50%; height: 50%; display: block; margin: 0 auto;"></canvas>
    <script type="text/javascript" src="es6-promise.js"></script>
    <script type="text/javascript" src="browserfs.min.js"></script>
    <script type="text/javascript" src="loader.js"></script>
    <script type="text/javascript">
      var emulator = new Emulator(document.querySelector("#canvas"),
                                  null,
                                  new JSMAMELoader(JSMAMELoader.driver("targ"),
                                                   JSMAMELoader.nativeResolution(256, 256),
                                                   JSMAMELoader.scale(3),
                                                   JSMAMELoader.emulatorJS("emulators/jsmess/mameexidy.js"),
                                                   JSMAMELoader.mountFile("targ.zip",
                                                                          JSMAMELoader.fetchFile("Game File",
                                                                                                 "examples/targ.zip"))))
      emulator.start({ waitAfterDownloading: true });
    </script>
  </body>
</html>

Before we continue, we will need to grab the relevant .js.gz for the game we are emulating from an Emularity archive. For me, this was mamebublbobl.js.gz. Direct Download. Run sudo gzip -d mamebublbobl.js.gz to extract the .js file needed to emulate this ROM.

The .js file extracted needs to be placed within the relevant directory specified within example_arcade.html.

To adjust this to my system, I need to edit driver, emulatorJS, mountFile, and fetchFile. If needed, the file can be adjusted further to suit your preferences or ROMS. My working example_arcade.html is seen below

<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
    <title>example arcade game</title>
  </head>
  <body>
    <canvas id="canvas" style="width: 50%; height: 50%; display: block; margin: 0 auto;"></canvas>
    <script type="text/javascript" src="es6-promise.js"></script>
    <script type="text/javascript" src="browserfs.min.js"></script>
    <script type="text/javascript" src="loader.js"></script>
    <script type="text/javascript">
      var emulator = new Emulator(document.querySelector("#canvas"),
                                  null,
                                  new JSMAMELoader(JSMAMELoader.driver("bublbobl"),
                                                   JSMAMELoader.nativeResolution(256, 256),
                                                   JSMAMELoader.scale(3),
                                                   JSMAMELoader.emulatorJS("emulators/mamebublbobl.js"),
                                                   JSMAMELoader.mountFile("bublbobl.zip",
                                                                          JSMAMELoader.fetchFile("Game File",
                                                                                                 "examples/bublbobl.zip"))))
      emulator.start({ waitAfterDownloading: true });
    </script>
  </body>
</html>

Found on Emularity's GitHub, loader.js around line 1228, we can see the Emulator object's source code below.

   /**
    * Emulator
    */
   function Emulator(canvas, callbacks, loadFiles) {
     if (typeof callbacks !== 'object') {
       callbacks = { before_emulator: null,
                     before_run: callbacks };
     }
     var js_url;
     var requests = [];
     var drawloadingtimer;
     // TODO: Have an enum value that communicates the current state of the emulator, e.g. 'initializing', 'loading', 'running'.
     var has_started = false;
     var loading = false;
     var defaultSplashColors = { foreground: 'white',
                                 background: 'black',
                                 failure: 'red' };
     var splash = { loading_text: "",
                    spinning: true,
                    finished_loading: false,
                    colors: defaultSplashColors,
                    table: null,
                    splashimg: new Image() };

     var runner;

     var muted = false;
     var SDL_PauseAudio;
     this.isMuted = function () { return muted; };
     this.mute = function () { return this.setMute(true); };
     this.unmute = function () { return this.setMute(false); };
     this.toggleMute = function () { return this.setMute(!muted); };
     this.setMute = function (state) {
       muted = state;
       if (runner) {
         if (state) {
           runner.mute();
         } else {
           runner.unmute();
         }
       }
       else {
         try {
           if (!SDL_PauseAudio)
             SDL_PauseAudio = Module.cwrap('SDL_PauseAudio', '', ['number']);
           SDL_PauseAudio(state);
         } catch (x) {
           console.log("Unable to change audio state:", x);
         }
       }
       return this;
     };

Configure NGINX

If not installed, sudo apt install nginx to install and enable the nginx service on your local machine. If things are working normally, we should be able to visit localhost within a web browser to view the default nginx HTML example page. Next, we'll change this landing page to point to a directory where we will later copy our compiled emscripten files along with the relative emularity configurations.

Edit /etc/nginx/sites-enabled/default to pass to your preferred directory on your server by changing the root directive within the server block to point to your directoy, as seen below.

server {
        listen 80 default_server;
        listen [::]:80 default_server;
        root /var/www/mame-ems;
        server_name _;
        location / {
                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.
                try_files $uri $uri/ =404;
        }
}

From the directory where we ran emmake to compile mame using emscripten (the root directory of mame on our system), we will need to relocate the following files to the root NGINX directory

Additionally, if you have any other software files you would like to run with the MAME driver, be sure to move them to the root NGINX directory as well.

Documentation Generators

Documentation Generators

Read The Docs

Features

Static Generators

Read The Docs (RTD) is a documentation generator that utilizes either Sphinx or MkDocs to generate the documentation site and theme, depending on your choice. The RTD GitHub houses the configurations and files that generates the Official Website. This provides a good example of the many features RTD offers, and allows you to step through configurations in a more feature-rich and complete setup.

Since there are different choices for the backend static site generator, the deployment procedure is different depending on which one you choose. Below, we will cover deploying Read The Docs with Sphinx and MkDocs. These two sections are each entirely different deployments of RTD, where the difference is only the Backend Static Site Generator that is used to build your documentation.

Alternatively, you can simply sign up on the RTD official website and allow them to host the documentation for you by linking a repository to your user account and following the Quick Start Instructions. Since this is nearly turn-key, we won't cover deploying RTD this way.

Continuous Version Based Documentation

RTD utilizes webhooks within your VCS (GitHub, GitLab, BitBucket) to automatically build and deploy version-based documentation for your projects. If you've deployed a project to RTD, it automatically creates these webhooks for you. See the Webhook and Automation documentation for more info.

There are also two independent versions of RTD, one community and one for business related documentation that also supports private repositories and other security features. When visiting https://readthedocs.org, we are viewing community version. When visiting https://readthedocs.com, we are viewing the business version.

For the below documentation, we will be using the Community Version of Read The Docs since the business version is Subscription Based.

Downloadable Documentation

RTD supports downloading any hosted documentation in PDF, ePub, and Zipped HTML formats.

Need a place to host your Sphinx docs? readthedocs.org hosts a lot of Sphinx docs already, and integrates well with projects' source control. It also features a powerful built-in search that exceeds the possibilities of Sphinx' JavaScript-based offline search. - Sphinx

RTD adds search functionality to your documentation that, in comparison to the sphinx builtin search capabilities, is a noteworthy improvement.

RTD with MkDocs

Deploying RTD with MkDocs is as simple as first deploying a MkDocs instance, and then importing this documentation into RTD. The RTD documentation provides instructions for Getting Started with MkDocs, which outlines the process below.

Install MkDocs

MkDocs supports Python versions 3.5, 3.6, 3.7, 3.8, and pypy3. If you have and use a package manager (such as apt-get, dnf, homebrew, yum, chocolatey, etc.) to install packages on your system, then you may want to search for a "MkDocs" package. If your package manager does not have a recent "MkDocs" package, you can still use your package manager to install "Python" and "pip". Then you can use pip to install MkDocs. - MkDocs Official Documentation

The quote above seems to suggest the preferred method for installation is pip, though it is possible to use your package manager. To make these instructions more portable, I'll only cover mkdocs installation using pip.

sudo apt update && sudo apt upgrade
sudo apt install python-pip -y
pip install --upgrade pip
sudo pip install mkdocs
MkDocs Man Pages

If needed, MkDocs provides man pages that can be installed with click-man by running the following commands

An explanation on why the man pages are not installed automatically with pip install mkdocs is described within the click-man GitHub README

pip install click-man
click-man --target path/to/man/pages mkdocs

Following the Man Pages Index Guidelines we can install the man pages to the appropriate section using the following command

click-man --target /usr/share/man/man1 mkdocs

Now we can see that the man pages for mkdocs have been installed and reference them as needed directly within our terminal -

Configuring MkDocs

MkDocs offers a range of custom configuration options which can be tailored to suit your needs aesthetically and in terms of functionality. To get started, run the following commands

mkdocs new my-project
cd my-project

This will have created a new directory relative to your CWD ./my-project generated two files within, ./my-project/mkdocs.yml and ./my-project/docs/index.md. For this section, we will be modifying mkdocs.yml

site_name: My Docs
nav:
  - home: index.md
theme: readthedocs

This adds a title for our documentation and says two things, we've created an ./my-project/docs/index.md that represents the home page content, and we want to use the ReadTheDocs theme for our instance of MkDocs. Now to apply these changes, we rebuild the documentation

mkdocs build

This generates raw HTML files within ./my-project/site, which will have the following file structure

These files do not need to be tracked by your VCS since they are generated, and if using git to track this MkDocs instance can be ignored with echo "site/" >> .gitignore

To update the generated file index with any removals or renamed pages, you'll need to run the following command

mkdocs build --clean

Once these files are updated, provided that you are working locally and hosting remotely, you can update your remote host with the new generated files -

mkdocs build
scp -r ./site user@host:/path/to/web/root

Alternatively, you can work directly on the host and serve the ./my-project/site directory directly.

Themes

Community Themes

MkDocs is very themeable, you'll notice with the screenshot below that the Windmil theme is quite different from the ReadTheDocs theme used on the RTD Documentation.

Plugins

Mkdocs supports plugins which extend the functionality in more specific use cases. Check the GitHub Wiki for a complete list of plugins.

This feature came from a request for multiple projects documentation within a single MkDocs instance - Issue #265. Once plugins were added to MkDocs, Spotify/mkdocs-monorepo-plugin provided a plugin to support this feature, allowing for multiple documentation folders within a single MkDocs. Spotify even hosts a Live Demo showcasing the results of using the plugin and has well-written instructions on how to use it.

Deploying MkDocs

The files generated with MkDocs can be deployed using either GitHub Pages, RTD, or a local webserver like NGINX or Apache. Below, we look at a simple NGINX configuration to serve MkDocs

user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events { }

http {
  include mime.types;

  # Redirect root domains
  server {
    listen 80;
    server_name domain.com www.domain.com;
    # Pass to SSL
    return 301 https://www.domain.com$request_uri;

  }

  # SSL
  server {
    server_name domain.com  www.domain.com;
    server_tokens off;
    listen 443 ssl;
    ssl_certificate /etc/letsencrypt/live/domain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/domain.com/privkey.pem;

    # Configure Server-wide logging
    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;


    # Pass to container
    location / {
      # Point NGINX to static site files
      root /path/to/my-project/site;
      include proxy_params;
    }

  }
}

As a final step, we restart the nginx service and navigate to our domain in a web browser.

sudo systemctl reload nginx

For demonstration, I've duplicated this page within my mkdocs instance, shown in the screenshot below

This MkDocs Demo

You can even search for keywords right out of the box with the builtin search field -

RTD with Sphinx

WIP