Discourse

This is for the Discourse instance at https://discourse.hive13.org that Hive13 uses for its forum / mailing list.

(This page is very much a work-in-progress. TODO: Find every TODO below.)

What is this?

Discourse is open-source message forum software. It behaves like most "traditional" forums: people post messages in topics, and replies to a topic form message threads. Users can also configure it to behave like a mailing list.

Hive13 self-hosts an instance of Discourse (rather than using a managed setup like at https://www.discourse.org/). This was set up in 2021, but not really transitioned until 2022. It was set up as a replacement for the Google Groups lists dating back to April 2009 when Hive13 was still the nascent "Cincinnati Hackerspace Project" (or CHP, or cincihackerspace), and all messages were migrated from these lists.

Most categories of this forum are publicly accessible and don't require an account to read. Posting messages requires setting up an account, instructions for which are given below.

How To Sign Up

Go to: https://discourse.hive13.org.

Click the "Log In" button, and then on the lower-right of the "Welcome back" pop-up, click "Create your account". (Or, if you're on a computer, the "Sign Up" button also works.)

This should get you to account creation:

Fill this out appropriately. If you use the same email address that you used on Hive13's older mailing list (on Google Groups), all of your existing posts there will be associated with the user you just created.

(If for whatever reason you made a Discourse account, but didn't use this email address to do so and you still want your old posts, ask us in the Site Support category and we can fix this. Likewise, if you forgot what email address you even used on the mailing list, ask us and we can look it up.)

Once you have signed up, you should see a message like this:

Discourse will send you an email to confirm your email address and activate your account; you should see something like this in your inbox:

Click the link, and then click the button to activate the account.

As the message should note, an admin still must approve this activated account - which may take us some time, as it is done manually. Discourse will send another email when this happens:

If this is taking too long, let one of us know in the Slack, e.g. the #hive13_it_infrastructure channel.

If you want to keep using the browser to interact with Discourse, nothing else is necessary.

If you want to read and post via email, you probably want mailing list mode. See the following section.

Enabling Mailing List Mode

In mailing list mode, Discourse will:

• Send you an email for every new post in a topic/thread. (That is: unless you mute the topic or the entire category.)
• Let you reply in this topic by replying to this email.
• Let you create a new topic by emailing the post to a specific address. Each category has its own email address for this:
  • General - discourse-reply+general@hive13.org
  • Leadership - discourse-reply+leadership@hive13.org
  • Wardens - discourse-reply+wardens@hive13.org
  • Site Support & Feedback (stuff for Discourse admins to see, account-related matters) - discourse-reply+support@hive13.org
  • Discourse Staff (admins/moderators only) - discourse-reply+staff@hive13.org
  • Sandbox (for testing bots and things, away from everyone else) - discourse-reply+sandbox@hive13.org

You can still use the browser to read and post. Mailing list mode doesn't force you to use email.

To enable mailing list mode, log into your Discourse account, and click your user-badge at the top-right to reveal 4 buttons:

Click the right-most button, and in menu below it, click "Preferences":

In the preferences page that opens up, click the "Emails" section on the left:

Find the "Enable mailing list mode" checkbox at the bottom, and click it. You should see a red note on top that mailing list mode has been enabled.

Finally, click the "Save Changes" button at the bottom. Look for the "Saved!" to the right of it.

To mute a topic or a category so that you don't receive emails for it: (TODO)

(See also: Using Discourse as a mailing list.)

Installing the App

If you want an app on your smartphone for Hive13's Discourse, we recommend its built-in Progressive Web App. Android and iOS both support this out of the box. (The "official" native app for Discourse is DiscourseHub on Android and iOS, but no one seems to like it, so we don't recommend it.)

Short instructions:

• Android: Browse to https://discourse.hive13.org/, press the three dots menu (more options / kebab menu) by the URL bar, find something like "Install" or "Install app", and confirm installing and adding to the Home screen.
• iOS: Browse to https://discourse.hive13.org/, press the "Share" button, find "Add to Home Screen", and confirm this.

Example screenshots are given below.

iPhone example

On an iPhone, it may look something like this (note the red boxes drawn to highlight where to press):

Android & Chrome example

On Android, it may look something like this (note the red boxes drawn to highlight where to press):

Android & Firefox example

For the rare ones among us using Firefox on Android, this still works fine:

Care and Feeding of the Discourse

(These are placeholders to be filled in later.)

Backups and Restoring

Where are backups?

This Discourse stores a few daily backups. You may access these backups directly at the Discourse if you are logged in as an admin.

This of course does nothing to help you if the entire Discourse instance is inoperable. The backups are also available at:

• /var/discourse/shared/standalone/backups/default on the VM itself.
• An Azure Storage share which the VM accesses via CIFS. They are automatically expired/rotated in a GFS scheme.

To see this share:

• Log into the admin account on Azure (credentials are in Bitwarden)
• Go to https://portal.azure.com
• Click "Storage accounts"
• Find "hivevmsnapshots"
• Find the "Data storage" section on the left, and "File shares" below this
• Find the "hivebackup" share

This link might work, or it might not. Microsoft likes making mazes.

The script /root/discourse_sync_backup.sh on the VM is responsible (see root's crontab) for syncing and rotating backups from what Discourse makes.

How do I restore a backup?

If you can access the Discourse (the Backups tab of the Admin view), you may restore backups from here - but you may first need to enable restoring. To enable restoring, go to the Settings tab of the Admin view, find Backups on the left, and find "allow restore".

If the Discourse is completely inaccessible, the most direct way is probably to create a new Discourse instance and then go through the above.

2022 August addendum on migration

'Leadership' category issues

In the initial setup, the Leadership and Warden categories were restricted to a respective "leadership" and "wardens" group. People pointed out that leadership@hive13.org originally would relay any email to the Leadership list to allow outside parties to communicate and start a message thread. To try to allow the same functionality, we enabled "Accept emails from anonymous users with no accounts".

This led to some problems like those noted in How to Use Discourse as a Private Support/Ticket System and in Private support categories: staged users can email in, but not registered users. In particular:

• The intention was that anyone could create a new topic in Leadership by emailing to the proper address. This worked properly for outsiders (i.e. those emailing from an address not associated with any Discourse user), and for Discourse users in the "leadership" group - but Discourse would reject posts coming from any email address associated with an existing user that was not in the "leadership" group, that is, literally every non-leadership user.
• Replies to outsiders were handled badly. If an outsider started a topic in Leadership, and someone replied there from the site, the outsider would receive an email which did not thread properly.
• In the same scenario, there was no easy way to remove the outsider from the recipients (e.g. so that leadership could discuss something internally first).
• In the same scenario, the outsider was receiving replies which contained other formatting and links to a Discourse topic which they could not even open (because the Leadership category is not publicly-readable). This also meant that if a leadership member replied to the outsider via email (i.e. in Discourse's mailing list mode), their reply would be mangled en route to the outsider.
• Along the same lines, signature blocks were being cut off of emails. This could be disabled with the "Trim incoming emails" setting, but this led to problems elsewhere in Discourse with people using mailing list mode: their replies were shown with the full verbatim contents of what they were replying to, including various banners and possibly internal email addresses - which was both obnoxious and a possible privacy issue.
• Incoming HTML messages were being mangled, both in the Discourse interface and in the emails it was relaying.

After seeing this and reading Start a New Topic via Email we tried experimenting with using a group inbox rather than a category. This solved the first issue above (non-leadership users could create topics), but failed to solve the others, and added some of its own issues:

• While it had some ability to change the recipients in a message thread, this became something stateful to the entire thread - rather than something per-message, like people expected (and like basically every email client of the past 3 decades handles it). That is, if leadership tried to remove a recipient from the thread so that they could discuss internally, Discourse would reject any messages from the recipient until they were added back.
• The interface felt like a kludge that was more of an annoyance to navigate compared with categories, despite being used similarly.
• Topics in the Leadership category would have required migration to personal messages, but there was no way to do this in bulk in the UI. While its API could probably do this, it was another added annoyance.

The current solution being proposed is to set up a simple mailing group (e.g. in GSuite) and then setting the Leadership category up as described in Creating a read-only mailing list mirror. In the tests done so far, it appears to solve every issue described above, but:

• HTML emails are still not rendered perfectly. (They are handled better, but still show some issues.)
• The category would be read-only; any interaction would have to be via email.
• Discourse doesn't show very well if a recipient was removed from a message.
• The mailing group must be manually kept in sync with the "leadership" group in Discourse (though perhaps there are ways to automate this).

Older migration notes

Migrated from: Intweb_and_Google_Groups_Migration_Thoughts

This needs cleaned up a bit. Anything of relevance in the to-do list below should be organized in another section and the rest should be removed.

To-do before migration

• Done: Make sure things like 'minimum trust level' are sensible to let new users create topics via email
• 99% done: Write up a Wiki page with some instructions (with screenshots, ideally) for how to enable mailing list mode for a user - something like this.
  • Discourse - How To Sign Up
  • This just needs some review
• Done: Make a purposely restricted number of categories. This was set to:
  • General (migrate from 'cincihackerspace' mailing list) - discourse-reply+general@hive13.org for creating new topic
  • Leadership (migrate from eponymous mailing list) - discourse-reply+leadership@hive13.org
  • Wardens (migrate from eponymous mailing list) - discourse-reply+wardens@hive13.org
  • Site Support & Feedback (stuff for Discourse admins to see, account-related matters) - discourse-reply+support@hive13.org
  • Discourse Staff (admins/moderators only) - discourse-reply+staff@hive13.org
  • Sandbox (for testing bots and things, away from everyone else) - discourse-reply+sandbox@hive13.org
• Done: Set up email addresses for creating topics in each category.
• Done: Figure out how to get people to see the email addresses for creating topics (if they're interacting via email)
  • Custom email footers in mailing list mode - I put the list of addresses here, and a related message in user_notifications.signup_after_approval.text_body_template.
• 99% done: Backups
  • Discourse is already generating backups: https://discourse.hive13.org/admin/backups
  • They're at /var/discourse/shared/standalone/backups/default on the VM
  • A daily cron job is copying them to Azure archive storage and using rotate-backups to do grandfather-father-son rotating backups
  • Still needed (or wanted by Hodapp): keeping local, non-cloud copies of these too
  • I should probably also save the YAML files in /var/discourse/containers
• Done: Migrate settings to use discourse.hive13.org, not discoursetest.hive13.org. (The DNS records are set up.)
  • The Slack app needs also this URL for the discourse-chat-integration plugin.
• Minor: Get Slack bot set up for #mailinglist channel.
  • Mostly solved. Get bot into correct channel and migrate URLs to 'real' URL.
• Done: Figure out the details of the final export/import.
  • For all future imports: rename 'cincihackerspace' folder to 'General'! The category was renamed, and ignoring this might lead to it re-importing 13 years of messages. (Later note: It may be a moot point because everything now goes to Uncategorized anyway. You'll have to move topics manually - so either keep the updates small, or go them one list at a time.)
  • Probably something like: Do one export/import right before the two-weeks message goes out (obviously, make sure Discourse is back up before sending message), and then another after the Google Groups lists are put into read-only mode.
  • There are three mailing lists: Hive13 Makerspace (cincihackerspace), Leadership, and Wardens.
  • Re-import properly ignores duplicates. I did run into a problem in which it failed to put Leadership messages into the category it created.
• Done: Write up transition post to send to Google Groups.
• Set up leadership@hive13.org and wardens@hive13.org accounts and forward to the respective topic-creation emails. (Also members@hive13.org to send to Leadership). Make sure Leadership allows non-users to post, like it does now - even if it requires moderation/approval.
• Send that final Google Groups post (to all mailing lists), export the final mbox (likewise), and turn off those mailing lists!

To-do, but not a migration requirement

• Per-category niceties:
  • Set good descriptions
  • Set up topic templates (where appropriate)
  • Set up sensible colors and banners/logos
  • Tune permissions
• Per-group niceties:
  • Avatars, colors, descriptions
• Set up some tag groups that are meaningful; for instance:
  • Votes (restricted to members only)
  • Officers or leadership, with tags inside for titles like "president" or "board", restricted to who holds that office
  • Wardens, restricted likewise
  • All else should be pretty free-form for now
• Update references on the website (and perhaps Wiki) to refer to the Discourse (or the "forum", perhaps - what do we want?)
  • Any links need to be updated, of course

Migration notes & problems

This section is some more in-depth notes, even on things that are already solved. The prior section has the actual to-do items.

• TODO also get some notes on:
  • URL
  • Setup procedure, including integrations, and especially things that weren't obvious
  • Where backups are stored
• Integrations:
  • Mostly solved - Email
    • Current state: SMTP for outgoing emails is set up (via sendgrid). Reply-via-email is set up via POP3. I just verified via mailing-list mode that this works both for replying via email, and creating new topics via email.
    • Can we assign tags to a topic that is started via email?
    • Creating new topics via email now works. Each category has a "Custom incoming email address:" field which needs set to something we can receive on. I verified that it works for "discourse-reply+itinfra@hive13.org" with our existing account (just give it something unique after the +), but of course if we want a nicer address we could make another gmail account.
    • The POP3 way was done with Set up Reply via Email Support for discourse-reply@hive13.org (creds are in the Bitwarden).
    • "mailing list mode" mysteriously deactivated - likely the culprit for why it did not work prior.
    • Misc links of note: Direct-delivery incoming email for self-hosted sites, Set up Reply via Email Support
    • Reply via email can be via POP3 or pushing via the Discourse API.
    • For the Discourse API way, this look like it is made for something like https://github.com/discourse/mail-receiver - which runs Postfix and pushes every received email to Discourse via this API, and Discourse mostly handles the spam issue by checking against valid senders/receivers.
    • I am guessing that this latter way would require MX records for DNS. (Note that discoursetest.hive13.org isn't our 'final' domain for this, so we may need to change this when we go live).
    • I think that mail-receiver handles only receiving, not sending, and so we can keep using sendgrid's SMTP for this (and, I think, avoid the complexity of trying to run our own MTA).
  • Mostly solved - Slack and various other chat systems:
    • See Chatroom Integration Plugin (discourse-chat-integration)
    • https://github.com/discourse/discourse-chat-integration is the repo; it follows the generic plugin instructions
    • Slack integration
    • Progress so far: Hodapp has the bot set up and was able to use /discourse commands to set up notifications... but for some reason at the configuring rules step is not able to see this table for configuring rules. See also: https://hive13.slack.com/apps/A02MMCY7P41-discourse?tab=more_info
  • Unsolved (but not critical) - MediaWiki: See https://www.mediawiki.org/wiki/Special:MyLanguage/Extension:Discourse maybe
• Solved - How do we migrate content from Google Groups?
  • Hodapp had success with exporting an mbox from Google Takeout, then using this method - the regular import, not Google Groups import. (Trying to import directly from Google failed with cryptic errors about cookies, and another way of getting an mbox suggests that the failure was due to Google changing their lack-of-an-API to no longer be compatible.)
  • The cincihackerspace mailing list data was around 3.2 GB - 2.9 GB in an mbox file, and a 300 MB tarball with what looks like group/member metadata.
  • The cliget addon was extremely useful for this, as it let me download the file onto the VM rather than requiring me to download it all locally and then re-upload it.
  • This is a slow process. Google Takeout takes how ever long it feels like (hours? days?), plus downloading the file into the right place. Discourse import takes about 2 hours. "/var/discourse/launcher rebuild app" was necessary afterwards; I am guessing this is because it had to clear out some old containers.
• Mostly solved - How do we migrate users from Google Groups? (How much can we automate?)
  • The import script from the link above creates "staged users" by default for users & email addresses seen in the imported messages; it can be configured to create "regular users" instead.
  • For now, it looks like we'll just let users make accounts and they'll need manual activation.

Draft message for 2 weeks before

Subject: This Google Group will migrate to Discourse on May 17, 2022.

Hive13 has used Google Groups for its mailing lists since 2009 - however, on May 17, 2022, these will all be turned to archive-only, and everything migrated to a self-hosted Discourse instance at https://discourse.hive13.org/.

Questions:

In the fewest words possible, what do I need to do?

• If you read from your web browser: Go to https://discourse.hive13.org/ instead of https://groups.google.com.
• If you want to be able to post: Create a Discourse account by following: https://wiki.hive13.org/view/Discourse#How_To_Sign_Up
• If you are used to getting messages as emails: https://wiki.hive13.org/view/Discourse#Enabling_Mailing_List_Mode

What is Discourse?

• Discourse is message forum software. It behaves like most "traditional" forums: people post messages in topics, and replies to a topic form message threads. Users can also configure it to behave like a mailing list.

What do I need to do? Do I need to sign up?

• You do *not* need an account at this Discourse to read messages. Just browse to https://discourse.hive13.org/.
• If you want to post, you'll need to create an account. See the steps at Discourse - How To Sign Up.
• You can create an account anytime, including now. We must manually approve new accounts, just like the Google Groups.
• If you want older posts you made on the Google Groups to still be associated with your Discourse account, use the same email address that you used on the mailing list. (If you forget, we can correct this later - just ask.)
• By default, this does not behave as a mailing list - but you can enable Mailing List Mode for your account, i.e. you will receive emails for every new message that is posted, and you may send emails to reply to an existing thread or to create a new topic. See https://wiki.hive13.org/view/Discourse#Enabling_Mailing_List_Mode to set this up. Note that, unfortunately, Discourse does *not* support a digest mode.

Where do I go for help on Discourse itself, complaints, suggestions, feedback, and so on?

• If you've signed up already, use the Site Support & Feedback category in the Discourse.
• You can also reply here in this message thread (if the group isn't archived yet).
• You can ask in #mailinglist on Slack.
• Failing all else, you can email Chris (hodapp87@hive13.org) or Heath (heath@hive13.org).

Where are all the old messages?

• All content from the groups, including images and other attachments, was imported into the Discourse. We'll migrate newer content, but it may take some time as it's not automatic.
• The Google Groups pages will remain online, but in read-only mode. We have no plans of deleting these groups or their content - though Google might.

Why was this done?

• The ecosystem of products and companies changed considerably since we chose Google Groups in 2009. Google has had Google Groups merely on life-support for many years now. We'd rather move everything now while we have the time to do it at our own pace. Some nice integrations that we had in the past - such as integrating the mailing list with Slack - are now gone because Google removed the support.
• We've received a regular stream of complaints about Google Groups being very annoying to interact with on phones.
• Moving to a self-hosted Discourse instance means that we have full control of our own data.
• Because Discourse is quite a nice piece of software, and we have plans of using its integrations with our Wiki, our Slack instance, and possibly our GitHub.

Does this replace Slack? Does this replace the Wiki?

• No - it just replaces the Google Groups.

What does self-hosted mean?

• It means that Hive13 hosts and manages its own private instance of the Discourse software. In our case, it is self-hosted on an Azure virtual machine, paid for with free Azure credits Hive13 receives for being a nonprofit - but if Azure shut this off tomorrow, we could run it on any other Linux server.
• "Self-hosted" is in contrast to a managed Discourse instance, e.g. at https://www.discourse.org/.
• In both cases, it is powered by open-source software: https://github.com/discourse/discourse

Why not shut the mailing list down completely, and instead solely use Slack/Discord/reddit/Facebook/Matrix/IRC/whatever?

• With regards to most of those: because we prefer to avoid proprietary walled-garden platforms which leave other companies in charge of our data, our user accounts, and on what is "allowed" content on their platform. We also don't use your data for tracking or advertising, and we'd prefer to avoid handing it over to services that do.
• With regards to real-time ones like Slack and Discord: because we think it's important to still have a communication medium that is slower, asynchronous, longer-form, and more durable.
• Further, because we still have some number of users that prefer to interact via email.