added posts
@@ -1,155 +1,58 @@
|
||||
+++
|
||||
title = "A Fresh Coat of Paint"
|
||||
description = "It's time for something new."
|
||||
url = "posts/a-fresh-coat-of-paint"
|
||||
date = 2021-12-31T21:04:00-08:00
|
||||
title = "A Fresh Coat Of Paint"
|
||||
description = "Migrating to Zola"
|
||||
author = "Ian Andwati"
|
||||
date = 2023-08-24
|
||||
[taxonomies]
|
||||
tags = ["webdev", "CSS", "HTML"]
|
||||
tags = ["zola", "blog", "tera", "rust"]
|
||||
+++
|
||||
I'm starting the new year with a new job. To paraphrase a friend, "it's just
|
||||
moving from one `$BIGCORP` to another", but it's still exciting. I worked my
|
||||
last gig for 5 years, so I'm nervous, but also very ready to do something new.
|
||||
While I'm doing one new thing I might as well do another. Taking some time off
|
||||
between jobs has given me enough breathing room to redo my website.
|
||||
|
||||
Using Zola as my preferred static site generator
|
||||
|
||||
<!-- more -->
|
||||
After using Hugo for over eight months I got tired of it and looked for other alternatives. That's when I stumbled upon Zola. Zola is a static site generator (SSG), similar to [Hugo](https://gohugo.io/), [Pelican](https://blog.getpelican.com/), and [Jekyll](https://jekyllrb.com/).
|
||||
|
||||
## New Features
|
||||
It is written in Rust and uses the Tera template engine, similar to Jinja2, Django templates, Liquid, and Twig. SSGs use dynamic templates to transform content into static HTML pages. Static sites are thus very fast and require no databases, making them easy to host.
|
||||
|
||||
If you've been here before you'll probably have noticed a significant visual
|
||||
overhaul. The site is now in dark mode, has a more varied color palette, and is
|
||||
more responsive to differently sized viewports.
|
||||
My previous Hugo site used a theme because I wasn't really ready to learn Go's template engine. I'm already familiar with Python Django and its template system so using Tera wasn't hard because it's similar to jinja2 and the Django template syntax.
|
||||
|
||||
In addition to the visual changes on this site there are new features as well!
|
||||
Content is written in [CommonMark](https://commonmark.org/), a strongly defined, highly compatible specification of [Markdown](https://www.markdownguide.org/). Zola uses [pulldown-cmark](https://github.com/raphlinus/pulldown-cmark#pulldown-cmark) to parse markdown files. The goal of this library is 100% compliance with the CommonMark spec. It adds a few additional features such as parsing footnotes, Github-flavored tables, Github-flavored task lists, and strikethrough.
|
||||
|
||||
1. [Posts] now have summaries thanks to Zola's [summary] feature, which allows
|
||||
you to use any content before a
|
||||
`<!-- more -->`
|
||||
comment in a page's Markdown.
|
||||
Zola comes with built-in syntax highlighting, but you must first enable it in the configuration file.
|
||||
|
||||
2. There are [tags] as well! These were actually always there, but I only put
|
||||
them in post [front matter] and didn't expose them on any pages. Tags are
|
||||
supported via Zola's [taxonomies], which are much more complicated and
|
||||
powerful than simple tags demand.
|
||||
Zola is very minimalistic, it is installed only as a single binary, with no extra dependencies needed. It is configured in `toml` and you only use what is needed. For example here is the configuration for this site `config.toml`:
|
||||
|
||||
3. The metadata for site pages now includes [Open Graph protocol] and
|
||||
[Twitter card] support for a better display in social media.
|
||||
```toml
|
||||
title = "Ian Andwati"
|
||||
description = "Ian Andwati's personal website"
|
||||
base_url = "https://andwati.github.io"
|
||||
|
||||
## What I Learned
|
||||
generate_categories_pages = false
|
||||
generate_tags_pages = false
|
||||
generate_feed = true
|
||||
feed_filename = "rss.xml"
|
||||
minify_html = false
|
||||
taxonomies = [{ name = 'tags', feed = false }]
|
||||
compile_sass = true
|
||||
|
||||
Every time I update this website I learn something new. I continued to use
|
||||
vanilla HTML and CSS and eschew JavaScript, but was still blown away by how
|
||||
little I know in the webdev space.
|
||||
[markdown]
|
||||
highlight_code = true
|
||||
highlight_theme = "gruvbox-dark"
|
||||
|
||||
In the interest of chronicling my newfound knowledge, here are a handful of the
|
||||
things I learned.
|
||||
[link_checker]
|
||||
|
||||
- There are a whole lot of [semantic] HTML tags! I was already using tags like
|
||||
`<article>`, `<header>`, and `<footer>`, but I just learned about the
|
||||
[`<time>`][time] tag. This allows human-readable dates to still be parsed by
|
||||
machines with precision.
|
||||
|
||||
```html
|
||||
<time datetime="1970-01-01T00:00:00+00:00">January 1st, 1970</time>
|
||||
```
|
||||
|
||||
I used it for post metadata where I display only the year, month, and day,
|
||||
but put the full [ISO-formatted] publication date in the `datetime`
|
||||
attribute.
|
||||
|
||||
- CSS has variables! This isn't really new, but it's still cool. I had been
|
||||
relying on [Sass] to keep my CSS [DRY], but I committed more fully to vanilla
|
||||
CSS this go around and was happy to use this feature.
|
||||
|
||||
For example, with this site, I defined colors in the [`:root`][css-root]
|
||||
pseudo-class so I could reference them later using [`var`][var].
|
||||
|
||||
```css
|
||||
:root {
|
||||
--background-color: #1d2021;
|
||||
--foreground-color: #ebdbb2;
|
||||
}
|
||||
|
||||
body {
|
||||
background-color: var(--background-color);
|
||||
color: var(--foreground-color);
|
||||
}
|
||||
```
|
||||
|
||||
- There's now a [media query] for [colorscheme][prefers-colorscheme]. I
|
||||
ultimately decided not to use it because I wanted a dark website, but I'm
|
||||
glad it exists. Building on the above CSS, you could change background and
|
||||
foreground color based on user preference.
|
||||
|
||||
```css
|
||||
@media (prefers-color-scheme: light) {
|
||||
:root {
|
||||
--background-color: #f9f5d7;
|
||||
--foreground-color: #3c3836;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- CSS selectors, especially pseudo-class functions, are extremely powerful.
|
||||
Here's the most complex CSS currently used by this site.
|
||||
|
||||
```css
|
||||
/* When internal links are linked directly, display with the accent color. */
|
||||
:where(h2, h3, h4, h5, h6):target > a[href^="#"]::before {
|
||||
opacity: 1;
|
||||
color: var(--primary-accent-color);
|
||||
}
|
||||
```
|
||||
|
||||
It uses the [`:where()`][css-where] function and an [attribute selector] to
|
||||
ensure that all subheadings display an otherwise invisible element when linked
|
||||
to directly, but only when they're internal links! Being able to specify that
|
||||
kind of logic in a declarative form is neat stuff.
|
||||
|
||||
- Invisible whitespace in your HTML continues to be an enormous source of pain.
|
||||
My anchor tag template (the thing that makes the `#` display to the left of
|
||||
headings when you hover over them) had a newline at the end of the file. The
|
||||
newline was interpolated into the output as an extra node, which caused all
|
||||
headings to subtly shift right. I nearly tore my hair out fixing that bug.
|
||||
|
||||
I've no doubt that I've committed some grave CSS sins with this revamp.
|
||||
Nevertheless, I enjoyed seeing what's possible and the important part is that
|
||||
the site works. 😅
|
||||
[extra]
|
||||
author = "Ian Andwati"
|
||||
```
|
||||
|
||||
## Inspiration
|
||||
|
||||
I would be remiss not to mention the people who inpired these changes. The
|
||||
colorscheme uses a subset of [Pavel Pertsev]'s [gruvbox], which I've used as my
|
||||
syntax highlighting theme for years. Post metadata was modeled after [Alexis
|
||||
King]'s site. I borrowed ideas for the Open Graph and Twitter card support from
|
||||
[Andrew Kvalheim] and [Amos Wenger].
|
||||
The motivation to use Zola also came from my decision to move from independently developed themes. I decided to write the HTML and CSS myself and endeavor to maintain the site individually. The base CSS is borrowed from [Reilly Tucker Siemens](https://tuckersiemens.com/) who had created a nice gruvbox palette for his site.
|
||||
|
||||
[Ruud van Asseldonk] deserves the lion's share of credit for these changes. I
|
||||
spent **hours** learning from their [meticulously crafted CSS]. This update
|
||||
wouldn't have been possible without everything I learned from them.
|
||||
I improved on it by adding an archive page, pagination, a comment system, and Google Analytics (more of these in later posts). I also implemented article read time and word count.
|
||||
|
||||
[Posts]: /posts
|
||||
[summary]: https://www.getzola.org/documentation/content/page/#summary
|
||||
[tags]: /tags
|
||||
[front matter]: https://www.getzola.org/documentation/content/page/#front-matter
|
||||
[taxonomies]: https://www.getzola.org/documentation/content/taxonomies/
|
||||
[Open Graph protocol]: https://ogp.me/
|
||||
[Twitter card]: https://developer.twitter.com/en/docs/twitter-for-websites/cards/overview/markup
|
||||
[semantic]: https://developer.mozilla.org/en-US/docs/Glossary/semantics
|
||||
[time]: https://developer.mozilla.org/en-US/docs/Web/HTML/Element/time
|
||||
[ISO-formatted]: https://en.wikipedia.org/wiki/ISO_8601
|
||||
[Sass]: https://sass-lang.com/
|
||||
[DRY]: https://en.wikipedia.org/wiki/Don%27t_repeat_yourself
|
||||
[css-root]: https://developer.mozilla.org/en-US/docs/Web/CSS/:root
|
||||
[var]: https://developer.mozilla.org/en-US/docs/Web/CSS/var()
|
||||
[media query]: https://developer.mozilla.org/en-US/docs/Web/CSS/Media_Queries/Using_media_queries
|
||||
[prefers-colorscheme]: https://developer.mozilla.org/en-US/docs/Web/CSS/@media/prefers-color-scheme
|
||||
[css-where]: https://developer.mozilla.org/en-US/docs/Web/CSS/:where
|
||||
[attribute selector]: https://developer.mozilla.org/en-US/docs/Web/CSS/Attribute_selectors
|
||||
[Pavel Pertsev]: https://github.com/morhetz
|
||||
[gruvbox]: https://github.com/morhetz/gruvbox
|
||||
[Alexis King]: https://lexi-lambda.github.io/
|
||||
[Andrew Kvalheim]: https://andrew.kvalhe.im/
|
||||
[Amos Wenger]: https://fasterthanli.me/
|
||||
[Ruud van Asseldonk]: https://ruudvanasseldonk.com/
|
||||
[meticulously crafted CSS]: https://github.com/ruuda/blog/blob/e02b663d76dc882952aec37319cafde221695ada/templates/page.css
|
||||
## What next
|
||||
|
||||
The site is a work in progress and is far from polished. I plan on implementing a newsletter to notify readers of new posts. The site is primarily a dark theme, in the future I'll also add theme toggling and maybe different palettes.
|
||||
|
||||
You can subscribe to the site's RSS feed at [https://andwati.github.io/rss.xml](https://andwati.github.io/rss.xml).
|
||||
@@ -1,345 +0,0 @@
|
||||
+++
|
||||
title = "Announcing Layabout"
|
||||
description = "Announcing Layabout, an event handler for the Slack RTM API."
|
||||
date = 2018-06-30T00:16:52-07:00
|
||||
[taxonomies]
|
||||
tags = ["Slack", "Python", "Layabout"]
|
||||
+++
|
||||
|
||||
Today I'm announcing [Layabout], my first official Python library. Layabout is
|
||||
a small event handling library on top of the
|
||||
[Slack Real Time Messaging (RTM) API][Slack RTM API]. You can get it right now
|
||||
on [PyPI].
|
||||
|
||||
<!-- more -->
|
||||
|
||||
## What's It Good For?
|
||||
|
||||
You can think of Layabout as a micro framework for building Slack bots. Since
|
||||
it wraps Slack’s RTM API it does best with tasks like interacting with users,
|
||||
responding to channel messages, and monitoring events. If you want more ideas
|
||||
on what you can do with it keep reading or check out the [examples].
|
||||
|
||||
## Why?
|
||||
|
||||
Why choose Layabout when the [Slack Events API] exists and there's already an
|
||||
officially supported [events library]? If these points resonate with you then
|
||||
Layabout is for you.
|
||||
|
||||
- You don't want set up Flask or a similar framework.
|
||||
- You don't want to configure a reverse proxy.
|
||||
- You don't want to acquire an SSL certificate.
|
||||
- You don't want to do any of the myriad tasks associated with best practices
|
||||
in hosting a web app just to respond to Slack events.
|
||||
- You're satisfied with the large set of [events] supported by the
|
||||
[RTM API][Slack RTM API].
|
||||
|
||||
## Why Not?
|
||||
|
||||
Layabout won't be for everone and that's OK. If these points resonate with you
|
||||
then you probably _do_ want to use the official events library.
|
||||
|
||||
- You can't afford to have a persistent WebSocket connection in your
|
||||
application.
|
||||
|
||||
## A Practical Example
|
||||
|
||||
If you want to download it and start playing with it as you read the rest of
|
||||
this blog post you can install it by running
|
||||
|
||||
```bash
|
||||
pip install layabout
|
||||
```
|
||||
|
||||
Once you've got Layabout installed let's take a look at what it's capable of by
|
||||
borrowing the code example right from its
|
||||
`README.rst`.
|
||||
|
||||
|
||||
```python
|
||||
from pprint import pprint
|
||||
from layabout import Layabout
|
||||
|
||||
app = Layabout()
|
||||
|
||||
|
||||
@app.handle('*')
|
||||
def debug(slack, event):
|
||||
""" Pretty print every event seen by the app. """
|
||||
pprint(event)
|
||||
|
||||
|
||||
@app.handle('message')
|
||||
def echo(slack, event):
|
||||
""" Echo all messages seen by the app except our own. """
|
||||
if event.get('subtype') != 'bot_message':
|
||||
slack.rtm_send_message(event['channel'], event['text'])
|
||||
|
||||
|
||||
def someone_leaves(events):
|
||||
""" Return False if a member leaves, otherwise True. """
|
||||
return not any(e.get('type') == 'member_left_channel'
|
||||
for e in events)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
# Automatically load app token from $LAYABOUT_TOKEN and run!
|
||||
app.run(until=someone_leaves)
|
||||
print("Looks like someone left a channel!")
|
||||
```
|
||||
|
||||
In 28 lines of code we've used Layabout to do the following:
|
||||
- Register a `debug` handler that triggers on **all** RTM events to pretty
|
||||
print them.
|
||||
- Register an `echo` handler that triggers on `message` events to echo them
|
||||
back into the channel they came from (unless of course we generated them).
|
||||
- Load our application token from an environment variable (`$LAYABOUT_TOKEN` by
|
||||
default).
|
||||
- Connect to the Slack API.
|
||||
- Continuously listen for events, calling the appropriate handlers until
|
||||
`someone_leaves` a channel we have access to.
|
||||
|
||||
Now that we've looked at what Layabout is, why you might want to use it, and
|
||||
how to use it let's look a bit deeper into its design and implementation.
|
||||
|
||||
## Design
|
||||
|
||||
If you're familiar with the superb [Flask] library then Layabout probably looks
|
||||
eerily similar to you. That's no accident and hopefully [Armin Ronacher] thinks
|
||||
imitation is the sincerest form of flattery.
|
||||
|
||||
More concretely, I think Python decorators are a powerful combination of
|
||||
simplicity and flexibility. They also lend themselves particularly well to
|
||||
event-driven workflows.
|
||||
|
||||
The heart of Layabout is its aggressively simple
|
||||
[`Layabout.handle`][layabout-handle] method. Its normal invocation just
|
||||
guarantees the decorated function will accept a `SlackClient` and an event as
|
||||
arguments before registering it as a particular
|
||||
type of handler.
|
||||
|
||||
Having access to those two arguments alone opens up a wealth of possibilities.
|
||||
To maximize developer freedom I wanted to provide as thin a wrapper as I could
|
||||
on top of the already excellent [slackclient] library. Giving direct access to
|
||||
a `SlackClient` instance meant I didn't have to write my own functions for
|
||||
calling out to the RTM API and I could also take advantage of its ability to
|
||||
call the [Slack Web API] as well.
|
||||
|
||||
I also took inspiration from [pytest]'s
|
||||
[`pytest.mark.parametrize`][pytest-parametrize] decorator to give handlers more
|
||||
versatility by adding an extra `kwargs` parameter.
|
||||
|
||||
```python
|
||||
from layabout import Layabout
|
||||
|
||||
app = Layabout()
|
||||
|
||||
if __name__ == '__main__':
|
||||
name = input('What is your name? ')
|
||||
|
||||
@app.handle('hello', kwargs={'name': name})
|
||||
def hello(slack, event, name):
|
||||
print(f"Hello! My name is {name}.")
|
||||
|
||||
app.run() # Run forever.
|
||||
```
|
||||
|
||||
By adding a `kwargs` parameter we can not only use `Layabout.handle` as a
|
||||
decorator, but also to register functions at runtime with dynamic data. For
|
||||
example, this code logs events that happen, but only if they're in particular
|
||||
channels:
|
||||
|
||||
```python
|
||||
from layabout import Layabout
|
||||
|
||||
app = Layabout()
|
||||
|
||||
|
||||
def log_for_channels(slack, event, channels):
|
||||
""" Log the event if it happened in a channel we care about. """
|
||||
if event['channel'] in channels:
|
||||
print(f"{event['type']} happened in {event['channel']}!")
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
# A mapping of events to their respective channels.
|
||||
event_channels = (
|
||||
('star_added', ('G1A8FG8AE', 'C03QZSL29')),
|
||||
('star_removed' ('C47CSFJRK', 'C045BMR29', 'G13RTMGXY')),
|
||||
)
|
||||
|
||||
# For each event register a new handler for specific channels.
|
||||
for event, channels in event_channels:
|
||||
app.handle(event, kwargs={'channels': channels})(log_for_channels)
|
||||
|
||||
app.run() # Run forever.
|
||||
```
|
||||
|
||||
You could also use a [closure] or default arguments on a normal function
|
||||
definition for this and it might look a little cleaner, but for passing runtime
|
||||
data to a lot of functions those can be tedious options.
|
||||
|
||||
Ultimately, I tried to write a library that I would want to use. I'm more
|
||||
excited now than ever to work with Slack's APIs, so in that regard I think this
|
||||
library is already a success.
|
||||
|
||||
## Implementation
|
||||
|
||||
One of the hallmarks of this library is that it only supports Python 3.6+. I
|
||||
specifically chose to use only the most recent Python for three reasons:
|
||||
1. I wanted take advantage of all the new language features like type
|
||||
annotations, f-strings, better destructuring assignment, etc.
|
||||
2. I didn't want to limit myself to the least common denominator by worrying
|
||||
about backwards compatibility.
|
||||
3. [2020] is fast approaching, folks. Use Python 3 already. If you intend to
|
||||
keep Python as part of your stack you're rapidly running out of excuses not
|
||||
to modernize.
|
||||
|
||||
I normally try to drink as little of the Object Oriented Kool-Aid as possible,
|
||||
so I tried a functional approach first, but keeping track of what was going on
|
||||
with connection state with a class just made sense to me. It also ended up
|
||||
being cleaner to keep a handler registry on an instance. Since they're
|
||||
self-contained you could conceivably spawn multiple instances into their own
|
||||
threads/processes and run them all simultaneously if you're careful with your
|
||||
global mutable state.
|
||||
|
||||
### Async
|
||||
|
||||
Unfortunately I didn't see an easy way to use Python 3's
|
||||
[`async def`][async-def] because of the synchronous nature of `slackclient`'s
|
||||
`SlackClient.rtm_read` method. This is a Python 3 feature I'd really like to
|
||||
learn more about and event handling and async seem like a natural fit to me. If
|
||||
there's ever a reason to release a Layabout v2.0 I will probably push harder in
|
||||
this direction.
|
||||
|
||||
### Type Annotations
|
||||
|
||||
From a development stance, the best part about this entire project so far has
|
||||
been learning how to use Python 3 [type annotations]. I miss
|
||||
them whenever I'm working with a project that doesn't have them.
|
||||
|
||||
I did have one minor annoyance while working with type annotations. Layabout
|
||||
keeps an internal collection of all the event handlers that have been
|
||||
registered to it with this signature.
|
||||
|
||||
```python
|
||||
# Private type alias for the complex type of the handlers defaultdict.
|
||||
_Handlers = DefaultDict[str, List[Tuple[Callable, dict]]]
|
||||
```
|
||||
|
||||
I wanted to be even more restrictive and specify exactly what was required
|
||||
of the [`Callable`][callable] by defining a [type alias] for a `Handler`. The
|
||||
restrictions I sought to specify were:
|
||||
- A handler must take at least two positional arguments:
|
||||
- The first argument must be a `SlackClient`.
|
||||
- The second argument must be a dictionary of arbitrary types keyed by string
|
||||
(`Dict[str, Any]`) as that's what the Slack RTM API events are.
|
||||
- If the required arguments are satisfied a handler _may_ take any number of
|
||||
additional arguments of `Any` type.
|
||||
- A handler can return `Any` type.
|
||||
|
||||
I took a stab at expressing this as
|
||||
|
||||
```python
|
||||
Handler = Callable[[SlackClient, Dict[str, Any], ...], Any]
|
||||
```
|
||||
|
||||
Sadly, it would seem this doesn't work. Right now [mypy] complains with a
|
||||
|
||||
```
|
||||
error: Unexpected '...'
|
||||
```
|
||||
|
||||
I've been up and down the [Python typing project][python-typing], but even
|
||||
after visiting [issue #193][issue-193] and [issue #264][issue-264] can't find a
|
||||
simple syntax for expressing a function that has a minimum arity of two with
|
||||
required types, but is [variadic] thereafter and [generic] in the types it
|
||||
accepts.
|
||||
|
||||
There may, in fact, be a way to express this type with current annotations, but
|
||||
I haven't figured out what it is yet. It may also be the case that the
|
||||
difficulty in expressing this type is an indicator that a better API exists and
|
||||
should be preferred. For now I've settled on just declaring that a `Handler` is
|
||||
a `Callable`. I've got an auxiliary function that validates handlers to let
|
||||
users know if they've omitted a required positional argument.
|
||||
|
||||
Despite that small inconvenience type annotations are **awesome**! Go use them!
|
||||
I now firmly believe that supplemental static analysis makes for better
|
||||
software, even in dynamically typed languages.
|
||||
|
||||
### Run Method
|
||||
|
||||
As a final note on implementation, the [`Layabout.run`][layabout-run] method
|
||||
only has an `until` parameter because it made it **so** much easier for me to
|
||||
unit test. If you go read the [tests] you'll notice many of them get called with
|
||||
|
||||
```python
|
||||
layabout.run(until=lambda e: False)
|
||||
```
|
||||
|
||||
which saves me from the headache of trying to test an otherwise infinite loop.
|
||||
If `until` is `None` then `Layabout.run` just uses its own private function
|
||||
|
||||
```python
|
||||
def _forever(events: List[Dict[str, Any]]) -> bool: # pragma: no cover
|
||||
""" Run Layabout in an infinite loop. """
|
||||
return True
|
||||
```
|
||||
|
||||
Giving the looping conditional access to the events opened up enough
|
||||
possibilities that I decided to keep it as part of the design.
|
||||
|
||||
## Thanks
|
||||
|
||||
I want to extend a special thank you to [Alex LordThorsen], [Geoff Shannon],
|
||||
[Kyle Rader], and [Mike Canoy] for their help during the initial development of
|
||||
this library. In particular the feedback I got on [PR #2][pr-2] was incredible
|
||||
and radically changed the library for the better.
|
||||
|
||||
## What's Next?
|
||||
|
||||
If you're still here and Layabout sounds like fun to you then check out these
|
||||
links to get started.
|
||||
- 📜 [Documentation]
|
||||
- 🐛 [Bug Reports / Issues]
|
||||
- 💾 [Source Code]
|
||||
|
||||
I _happily_ entertain pull requests, so if something's not quite right feel
|
||||
free to jump in and submit your own fix if you're able. Happy Slacking!
|
||||
|
||||
[Layabout]: https://layabout.readthedocs.io/en/latest/
|
||||
[PyPI]: https://pypi.org/project/layabout
|
||||
[Slack RTM API]: https://api.slack.com/rtm
|
||||
[examples]: https://github.com/reillysiemens/layabout/tree/master/examples
|
||||
[Slack Events API]: https://api.slack.com/events-api
|
||||
[events library]: https://github.com/slackapi/python-slack-events-api
|
||||
[events]: https://api.slack.com/events
|
||||
[Flask]: http://flask.pocoo.org/
|
||||
[layabout-handle]: https://layabout.readthedocs.io/en/latest/api.html#layabout.Layabout.handle
|
||||
[closure]: https://en.wikipedia.org/wiki/Closure_(computer_programming)
|
||||
[slackclient]: https://github.com/slackapi/python-slackclient
|
||||
[Slack Web API]: https://api.slack.com/web
|
||||
[pytest]: https://docs.pytest.org/en/latest/
|
||||
[pytest-parametrize]: https://docs.pytest.org/en/latest/how-to/parametrize.html
|
||||
[Armin Ronacher]: http://lucumr.pocoo.org/about/
|
||||
[2020]: https://www.python.org/dev/peps/pep-0373/#update-april-2014
|
||||
[async-def]: https://docs.python.org/3/reference/compound_stmts.html#async-def
|
||||
[type annotations]: https://www.python.org/dev/peps/pep-0484/
|
||||
[type alias]: https://docs.python.org/3/library/typing.html#type-aliases
|
||||
[mypy]: https://github.com/python/mypy
|
||||
[python-typing]: https://github.com/python/typing
|
||||
[issue-193]: https://github.com/python/typing/issues/193
|
||||
[issue-264]: https://github.com/python/typing/issues/264
|
||||
[variadic]: https://en.wikipedia.org/wiki/Variadic_function
|
||||
[generic]: https://en.wikipedia.org/wiki/Generic_programming
|
||||
[callable]: https://docs.python.org/3/library/typing.html#callable
|
||||
[layabout-run]: https://layabout.readthedocs.io/en/latest/api.html#layabout.Layabout.run
|
||||
[tests]: https://github.com/reillysiemens/layabout/blob/ed617cdfec4ec31b681f51697f922d4979f83cb6/tests/test_layabout.py
|
||||
[Alex LordThorsen]: https://github.com/rawrgulmuffins
|
||||
[Geoff Shannon]: https://github.com/RadicalZephyr
|
||||
[Kyle Rader]: https://github.com/kyle-rader
|
||||
[Mike Canoy]: https://mikecanoy.net
|
||||
[pr-2]: https://github.com/reillysiemens/layabout/pull/2
|
||||
[Documentation]: https://layabout.readthedocs.io/en/latest/
|
||||
[Bug Reports / Issues]: https://github.com/reillysiemens/layabout/issues
|
||||
[Source Code]: https://github.com/reillysiemens/layabout/
|
||||
|
Before Width: | Height: | Size: 7.8 KiB |
|
Before Width: | Height: | Size: 9.2 KiB |
|
Before Width: | Height: | Size: 1.7 KiB |
|
Before Width: | Height: | Size: 6.5 KiB |
|
Before Width: | Height: | Size: 7.0 KiB |
|
Before Width: | Height: | Size: 5.7 KiB |
|
Before Width: | Height: | Size: 3.9 KiB |
|
Before Width: | Height: | Size: 5.6 KiB |
|
Before Width: | Height: | Size: 21 KiB |
|
Before Width: | Height: | Size: 19 KiB |
@@ -1,173 +0,0 @@
|
||||
+++
|
||||
title = "Deprecating Layabout"
|
||||
description = "Deprecating Layabout"
|
||||
date = 2019-12-31T17:29:00-08:00
|
||||
[taxonomies]
|
||||
tags = ["Slack", "Python", "Layabout"]
|
||||
+++
|
||||
Since [Layabout] launched last year it has been downloaded 5,755 times, gotten
|
||||
16 stars on GitHub, been used by a Portuguese startup to teach a
|
||||
[Haskell workshop], and received a [Twitter shout-out] from [@roach], one of
|
||||
the core contributors to the [official Python Slack client]. During that time the
|
||||
official client library also got a **lot** better! So much better, in fact,
|
||||
that I've decided to deprecate Layabout.
|
||||
|
||||
<!-- more -->
|
||||
|
||||
## When?
|
||||
|
||||
Layabout is officially deprecated on **January 1st, 2020** along with Python 2
|
||||
(_finally_ 😉). I'll be rolling out documentation changes and adding
|
||||
deprecation warnings in a final release within the next week or so. Technically
|
||||
Layabout will continue to function as long as the API interface supported by
|
||||
the 1.0 Slack library is supported, but you should transition off of it as soon
|
||||
as you are able. I recommend using the 2.0 Slack library instead.
|
||||
|
||||
## Why?
|
||||
|
||||
In May, [Rodney Urquhart] and contributors [rewrote the slackclient library]
|
||||
from the ground up in a way that closely matched some of the design decisions I
|
||||
made when creating Layabout. When I [announced Layabout] I included a quick
|
||||
example of using the framework to build an echo client that would repeat all
|
||||
messages in a given channel. To understand why Layabout no longer provides
|
||||
substantial benefit, let's compare that example written with Layabout to the
|
||||
same application using the 2.0 Slack client.
|
||||
|
||||
```python
|
||||
""" An echo client written using Layabout. """
|
||||
from layabout import Layabout
|
||||
|
||||
app = Layabout()
|
||||
|
||||
|
||||
@app.handle("message")
|
||||
def echo(slack: "slackclient.SlackClient", event: dict) -> None:
|
||||
""" Echo all messages seen by the app except our own. """
|
||||
if event.get("subtype") != "bot_message":
|
||||
slack.rtm_send_message(event["channel"], event["text"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
app.run()
|
||||
```
|
||||
|
||||
```python
|
||||
""" An echo client written using the official slackclient library. """
|
||||
import os
|
||||
from slack import RTMClient
|
||||
|
||||
|
||||
@RTMClient.run_on(event="message")
|
||||
async def echo(**payload) -> None:
|
||||
""" Echo all messages seen by the app except our own. """
|
||||
data = payload["data"]
|
||||
web_client = payload["web_client"]
|
||||
|
||||
if data["subtype"] != "bot_message":
|
||||
web_client.chat_postMessage(
|
||||
channel=data["channel"], text=data["text"], thread_ts=data["thread_ts"]
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
slack_token = os.environ["SLACK_API_TOKEN"]
|
||||
rtm_client = RTMClient(token=slack_token)
|
||||
rtm_client.start()
|
||||
```
|
||||
|
||||
The Layabout example is a little more terse, but neither is a lot of code. They
|
||||
both leverage a decorator pattern for callback registration and support type
|
||||
annotations. The official library also adds `async` support. It should be
|
||||
relatively straightforward to translate a Layabout application to using the
|
||||
offical library.
|
||||
|
||||
### Event-oriented decorator API
|
||||
|
||||
At the time that I wrote Layabout, the slackclient library provided more of a
|
||||
low-level API, so Layabout functioned as a refinement on reading from the
|
||||
WebSocket connection in a loop. It seems Rodney and I both agreed that the
|
||||
decorator pattern is much more convenient for callback registry. The only
|
||||
important thing that the official library does differently is give you a
|
||||
`payload` dictionary rather than individual arguments. Both a `slack` instance
|
||||
and an `event` are included in that `payload`, so this is a minor change.
|
||||
|
||||
### Async
|
||||
|
||||
The rewrite of the slackclient library brought with it clever, optional support
|
||||
for [Python's `async`/`await` syntax][async await]. At the time I was writing
|
||||
Layabout I wanted to add support for that myself, but the only available API
|
||||
was synchronous. Interacting with the Slack API via WebSockets lends itself
|
||||
naturally to an asynchonous approach. In my opinion, using the offical library
|
||||
will give a big advantage over Layabout here. If you've structured your
|
||||
Layabout callbacks well and haven't done necessarily synchronous operations,
|
||||
then you should be fine here.
|
||||
|
||||
### Type annotations
|
||||
|
||||
Like Layabout, the slackclient library now has [type annotations], which I
|
||||
think are a fantastic addition to Python. If you choose to add a type checker
|
||||
like [mypy] or [Pyre] to your toolkit, you can get the benefit of optional
|
||||
static analysis. Layabout no longer provides an extra service since this is
|
||||
supported by the slackclient library directly.
|
||||
|
||||
## What's Missing?
|
||||
|
||||
There are three main pieces of functionality that Layabout still provides that
|
||||
aren't available with the official slackclient library.
|
||||
|
||||
1. The `*` event
|
||||
|
||||
Layabout supported registering a function to trigger on **all** events with
|
||||
`@app.handle("*")`. This was primarily useful in debugging, but I think you
|
||||
can achieve similar results by setting the logging level to `logging.DEBUG`.
|
||||
|
||||
2. Auto-loading environment variables
|
||||
|
||||
Dealing with credentials is always a pain in the butt, so I designed
|
||||
Layabout to automatically load the Slack API token from `$LAYABOUT_TOKEN` by
|
||||
default. Or, with `app.run(connector=EnvVar("WHATEVER")` you could
|
||||
load the token from `$WHATEVER` environment variable you pleased.
|
||||
|
||||
3. The run `until` option
|
||||
|
||||
With Layabout you could use `app.run(until=...)` to provide an arbitary
|
||||
function which would trigger an exit from the application's core loop. I
|
||||
added this to enable cleaner testing with fewer mocks. However, bots are
|
||||
usually meant to be long-running applications, so I'm not sure this was ever
|
||||
useful enough to justify its existence.
|
||||
|
||||
These are all handy features, but I don't think any of them are consequential
|
||||
enough to be worth keeping Layabout around. If, going forward, you miss these
|
||||
features, they're probably better suited to be suggestions or pull requests to
|
||||
the official library now.
|
||||
|
||||
|
||||
## Thanks
|
||||
|
||||
If you used Layabout, _thank you_. I hope you got as much joy out of it as a
|
||||
user as I got in writing it. I would also like to thank these fine folks for
|
||||
their contributions in getting Layabout off the ground and helping maintain it.
|
||||
|
||||
- Alex LordThorsen ([@rawrgulmuffins])
|
||||
- Geoff Shannon ([@RadicalZephyr])
|
||||
- Kyle Rader ([@kyle-rader])
|
||||
- Mike Canoy ([@mikecanoy])
|
||||
- Sophie Anderson ([@SophieKAn])
|
||||
|
||||
[Layabout]: https://layabout.readthedocs.io/en/latest
|
||||
[Haskell workshop]: https://github.com/ricardojusto/haskell-workshop/blob/f96aa901700d0b10dad35f391a94017f502fb42a/s01/e10/bot/receive
|
||||
[Twitter shout-out]: https://twitter.com/roach/status/1019279698092744705
|
||||
[@roach]: https://twitter.com/roach
|
||||
[official Python Slack client]: https://github.com/slackapi/python-slackclient
|
||||
[Rodney Urquhart]: https://twitter.com/RodneyU215
|
||||
[rewrote the slackclient library]: https://slack.engineering/rewriting-the-slack-python-sdk-ea000f587de7
|
||||
[announced Layabout]: /posts/announcing-layabout
|
||||
[async await]: https://www.python.org/dev/peps/pep-0492/
|
||||
[type annotations]: https://www.python.org/dev/peps/pep-0484/
|
||||
[mypy]: http://mypy-lang.org/
|
||||
[Pyre]: https://pyre-check.org/
|
||||
[@rawrgulmuffins]: https://github.com/rawrgulmuffins
|
||||
[@RadicalZephyr]: https://github.com/RadicalZephyr
|
||||
[@kyle-rader]: https://github.com/kyle-rader
|
||||
[@mikecanoy]: https://github.com/mikecanoy
|
||||
[@SophieKAn]: https://github.com/SophieKAn
|
||||
@@ -0,0 +1,17 @@
|
||||
+++
|
||||
title = "File over app"
|
||||
description = "Take control of your artifacts"
|
||||
author = "Ian Andwati"
|
||||
draft = true
|
||||
date = 2023-12-18
|
||||
[taxonomies]
|
||||
tags = ["obsidian", "entropy", "notes"]
|
||||
+++
|
||||
|
||||
Take control of your artifacts
|
||||
|
||||
<!-- more -->
|
||||
|
||||
In my endless endevours to find the perfect note taking app led me to obsidian, a note taking software that uses markdown files as the file format. All software is ephemereal in nature and yoour artifacts need to survive the decline of your favorite app. The files you create should be available to you and shouldt rrequire specialized software to access.
|
||||
|
||||
Your data shouldt be locked behind a paywall or an authentication system that is not under your control.
|
||||
@@ -0,0 +1,41 @@
|
||||
+++
|
||||
title = "Django Secret Key Tutorial"
|
||||
description = "Managing secrets"
|
||||
author = "Ian Andwati"
|
||||
date = 2022-11-11T15:34:14.160Z
|
||||
[taxonomies]
|
||||
tags = ["tutorial", "django", "python", ".env"]
|
||||
+++
|
||||
|
||||
Managing the Django SECRET_KEY variable.
|
||||
|
||||
<!-- more -->
|
||||
|
||||
The Django `SECRET_KEY` variable is very crucial to your Django application. The secret key must be a large random value and it must be kept secret. Leaking this value to unauthorized people could lead to a security breach. The SECRET_KEY is used in Django for cryptographic signing. It is used to generate tokens and hashes, they can be recreated using this variable. If it is not configured Django throws a `django.core.exceptions.ImproperlyConfigured: The SECRET_KEY setting must not be empty` error
|
||||
|
||||
# Using Environment Variables
|
||||
|
||||
The secret key should not be committed to version control. It is best practice to store the value in a .env file which is added to the .gitignore file to un-track its changes. The values can be loaded programmatically into your settings.py file.
|
||||
|
||||
# Generating A New Secret Key
|
||||
|
||||
This solution is using python's secrets lib on the back
|
||||
|
||||
```python
|
||||
from django.core.management.utils import get_random_secret_key
|
||||
# print new random secret key
|
||||
print(get_random_secret_key())
|
||||
```
|
||||
|
||||
This code can be run in the terminal as a command:
|
||||
|
||||
```python
|
||||
python -c 'from django.core.management.utils import get_random_secret_key; \
|
||||
print(get_random_secret_key())'
|
||||
```
|
||||
|
||||
Alternatively, If you are using python 3.6+ then you can use the `secrets.token_hex(\[nbytes=None])` function:
|
||||
|
||||
```python
|
||||
python3 -c 'import secrets; print(secrets.token_hex(100))'
|
||||
```
|
||||
|
Before Width: | Height: | Size: 634 KiB |
@@ -1,308 +0,0 @@
|
||||
+++
|
||||
title = "git filter-wat"
|
||||
description = "Fighting git filter-branch and GPG."
|
||||
url = "posts/git-filter-wat"
|
||||
date = 2016-04-07T16:06:33-08:00
|
||||
[taxonomies]
|
||||
tags = ["GitHub", "PGP", "Git"]
|
||||
+++
|
||||
Welcome to this year's annual blog post!
|
||||
|
||||
I've been signing `git` commits for my [dotfiles] repository since its
|
||||
inception in October of last year, so I was excited to see that GitHub recently
|
||||
added [GPG signature verification]. All you have to do is upload your
|
||||
[public key] to GitHub and you'll be verifying commits like a champ. Or so I
|
||||
thought…
|
||||
|
||||
<!-- more -->
|
||||
|
||||
<figure>
|
||||
<img src='signature-doesnt-match-committer.png' alt="Signature Doesn't Match Committer">
|
||||
<figcaption>
|
||||
Unverified commits on GitHub. <a href='signature-doesnt-match-committer.png'>View full size.</a>
|
||||
</figcaption>
|
||||
</figure>
|
||||
|
||||
GitHub thinks I'm unverified. I think that's some baloney. I _know_ the public
|
||||
key I uploaded matches the private key I used to sign those commits. Oh, it
|
||||
looks like what they're really concerned about though is that the email on my
|
||||
PGP key doesn't match the email I used with `git`.
|
||||
|
||||
```
|
||||
commit 7d74300ee1cd9c2f17a39b143be331cad82fe464
|
||||
Author: Reilly Tucker Siemens <[email protected]>
|
||||
Date: Sat Mar 5 15:10:18 2016 -0800
|
||||
|
||||
Add cookiecutter configuration.
|
||||
|
||||
commit 672f175ba6db1be4f9714f7526c9ff6153c44a81
|
||||
Author: Reilly Tucker Siemens <[email protected]>
|
||||
Date: Sat Feb 20 14:20:34 2016 -0800
|
||||
|
||||
Add Go to PATH. Make Virtualenv Wrapper config more flexible.
|
||||
```
|
||||
|
||||
Now, before we go any further I should point out I wouldn't be having any
|
||||
problems at all if my `user.email` matched my GPG key to begin with, but
|
||||
hindsight is 20/20. As it stands, I have a problem and I need to fix it. I need
|
||||
to modify the authorship of these commits to match the email in my GPG key.
|
||||
|
||||
## How to Fix It
|
||||
|
||||
I recently learned that `git` has a tool called `git filter-branch` that can be
|
||||
used to make significant and otherwise tedious modifications to `git` history.
|
||||
A quick trip to Stack Overflow reveals I can use this tool to
|
||||
[change the authorship] of all the commits in my repository.
|
||||
|
||||
Reckless reading leads me to this potential solution.
|
||||
```bash
|
||||
git filter-branch -f --env-filter "GIT_AUTHOR_EMAIL='[email protected]'" HEAD
|
||||
```
|
||||
|
||||
<figure>
|
||||
<img src='git-filter-wat.png' alt='git-filter-wat'>
|
||||
<figcaption>
|
||||
A royally screwed up commit log. <a href='git-filter-wat.png'>View full size.</a>
|
||||
</figcaption>
|
||||
</figure>
|
||||
|
||||
|
||||
Oh. No. Something is _clearly_ wrong. Looks more like `git filter-wat`.
|
||||
|
||||
## How to Actually Fix It
|
||||
|
||||
To be honest, I half expected something like this to happen. With the previous
|
||||
command I was never asked to resign these commits. The text of my commit
|
||||
message has been clobbered with the PGP signature. How do I fix that?
|
||||
|
||||
Some advocate not signing individual commits and instead just
|
||||
[using a signed tag], but I don't subscribe to that idea. I really want each
|
||||
individual commit to be signed.
|
||||
|
||||
[Additional Stack Overflowing] (yes, that's a verb) indicates I can use
|
||||
`git filter-branch`'s `--msg-filter` and `--commit-filter` options to strip the
|
||||
PGP signature from the commit message and then resign each commit. This ends up
|
||||
looking like
|
||||
|
||||
```bash
|
||||
git filter-branch -f --env-filter "GIT_AUTHOR_EMAIL='[email protected]'" --msg-filter 'sed "/iQIcBA.*/,/.*END PGP SIGNATURE.*/d"' --commit-filter 'git commit-tree -S "$@"' HEAD
|
||||
```
|
||||
|
||||
The `--msg-filter` uses a `sed` expression to match and delete the PGP
|
||||
signature up to the `END PGP SIGNATURE` bit. This leaves the rest of the commit
|
||||
object (basically just the message) intact. The modified object is then passed
|
||||
to the `git commit-tree` in the `--commit-filter` which then requires me to
|
||||
resign the commit.
|
||||
|
||||
Annoyingly, when I actually ran this command I had to sign **each and every**
|
||||
commit even though I had a `gpg-agent` running. If anyone can tell me how to
|
||||
avoid that in the future I'd love to know. Luckily it was only 15 commits, but
|
||||
I would find entering a passphrase any more than that rather aggravating.
|
||||
|
||||
At this point the unwieldly `git filter-branch` incantation has been uttered.
|
||||
Let's just double-check the modified commits with a quick `git log`.
|
||||
|
||||
```
|
||||
commit 87051d659d16dbe037c9d61dbaaeea38e152a9ff
|
||||
Author: Reilly Tucker Siemens <[email protected]>
|
||||
Date: Sat Mar 5 15:10:18 2016 -0800
|
||||
|
||||
Add cookiecutter configuration.
|
||||
|
||||
commit c56aff44af574bca227587e0f12f5ce841afd2d0
|
||||
Author: Reilly Tucker Siemens <[email protected]>
|
||||
Date: Sat Feb 20 14:20:34 2016 -0800
|
||||
|
||||
Add Go to PATH. Make Virtualenv Wrapper config more flexible.
|
||||
```
|
||||
|
||||
**Looks good to me!** Notice that the email isn't the only thing that's
|
||||
changed. The commit hash is also completely different. These aren't modified
|
||||
commits, they're new git objects with the author, date, and commit message
|
||||
preserved. In order to get my changes up to GitHub I'll have to
|
||||
`git push --force origin master` to blow away my previous history. Most of the
|
||||
time this is probably a bad idea, but this repository exists just for me, so I
|
||||
feel comfortable taking the sledgehammer approach.
|
||||
|
||||
## Back to Square One
|
||||
|
||||
<figure>
|
||||
<img src='signature-doesnt-match-committer.png' alt="Signature Doesn't Match Committer">
|
||||
<figcaption>
|
||||
These commits are <em>still</em> unverified on GitHub. <a href='signature-doesnt-match-committer.png'>View full size.</a>
|
||||
</figcaption>
|
||||
</figure>
|
||||
|
||||
Well, that didn't change anything. What gives? What am I missing? How does
|
||||
GitHub expect this to work in the first place? New idea. What if I set my
|
||||
author email correctly from the get-go and create an entirely new signed
|
||||
commit?
|
||||
|
||||
```bash
|
||||
git config --global user.email [email protected]
|
||||
touch why-doesn\'t-this-work.txt
|
||||
git add why-doesn\'t-this-work.txt
|
||||
git commit -Sm "Why doesn't this work?"
|
||||
git push --force origin master
|
||||
```
|
||||
|
||||
<figure>
|
||||
<img src='oh-but-it-does.png' alt='Oh, but it does!'>
|
||||
<figcaption>
|
||||
Oh, but it <em>does</em> work! <a href='oh-but-it-does.png'>View full size.</a>
|
||||
</figcaption>
|
||||
</figure>
|
||||
|
||||
Now I think I'm crazy. This works, but _why_? Something **must** be different,
|
||||
but what is it? Let's check the `git log` again.
|
||||
|
||||
```
|
||||
commit 8f08abcef9f4126ca617b0247c52264a619b049c
|
||||
Author: Reilly Tucker Siemens <[email protected]>
|
||||
Date: Thu Apr 7 01:12:27 2016 -0700
|
||||
|
||||
Why doesn't this work?
|
||||
```
|
||||
|
||||
How does that look any different from the commit messages above that _didn't_
|
||||
work? It doesn't _appear_ to be any different, so let's take a deeper look.
|
||||
Borrowing from something I learned while reading the excellent [Git Immersion]
|
||||
tutorial I made use of `git cat-file` to inspect the commit objects. Running
|
||||
`git cat-file -p 87051d6` shows the commit object for a commit that GitHub
|
||||
won't verify.
|
||||
|
||||
```
|
||||
tree 8feaf4ea13ac445111d9213cd5f917085e381642
|
||||
parent c56aff44af574bca227587e0f12f5ce841afd2d0
|
||||
author Reilly Tucker Siemens <[email protected]> 1457219418 -0800
|
||||
committer Reilly Tucker Siemens <[email protected]> 1457219418 -0800
|
||||
gpgsig -----BEGIN PGP SIGNATURE-----
|
||||
Version: GnuPG v1
|
||||
|
||||
iQIcBAABAgAGBQJXBhWDAAoJEBtFjnx8sVSpqFoQAKAjWfQ7IfnnUHx/ZuBWdvQt
|
||||
cWt7+LMMmp6OgjATRv8QGoY6GDarLVMNZjhsvtfym5HWrdWk9WhtDqA9EbiLTdhD
|
||||
yFxWhIDVHjqWt5U7QkWWcIYDUhVJ/z8PShPfa9d0Vwjq+HPqziTwILYjoedqBqOr
|
||||
cce8sxSG1GppAuOyiYzqBoVfOC1ko+egh8gsl9pwrMO345dBp5ZMXtyxv4a4v/7s
|
||||
WhY2Ggf71EJ9YTWGBHe2FT8WEH5DjVZZpsFLRlO6BUklKf8PuUPDQVmpgx60L0qW
|
||||
MBmqcw1ftx3vwTAL/foxmE8KkMi5xnIPtUYDdo6d3a2ZeUuWDJBnb+ZxENTLl1DS
|
||||
rwYI/LKbJwZpMfegwnHaJFgBF7igM7poeP3pytN2qzXYRGyXkJPYYh8Di5/alaGt
|
||||
rp0rtlJJ2tk2m+V9MiqrO8HJoZrt1Y5z/Pg+Fo1yJdB+97fjYJfjnQ7+nTIgkfoB
|
||||
hQlIc/G+w194GEN1AO4P7CeCvXh3necNPKsUZw0BfXrRjEIKGb5Qrs31xY1AJtup
|
||||
8prxg4jV3EmKBzKng3E65QHTPAjQWl0FhdvI2Qd2ea+fpjSbTKRDSmdi8ghHb6C2
|
||||
Q1azXowhVOoqodINE7wc4OpDsc9hLzCUdY1z8iBgMzsYxjSwJGerRpAWT8mjmoK1
|
||||
Z1wjETnbqvu8FXuVgqpW
|
||||
=CYyg
|
||||
-----END PGP SIGNATURE-----
|
||||
|
||||
|
||||
Add cookiecutter configuration.
|
||||
```
|
||||
|
||||
Running `git cat-file -p 8f08abc` shows the commit object for a commit GitHub
|
||||
will verify.
|
||||
|
||||
```
|
||||
tree 4dd20b4bf0b143ef3b0ed73c7232b9cf3da669e5
|
||||
parent 7d74300ee1cd9c2f17a39b143be331cad82fe464
|
||||
author Reilly Tucker Siemens <[email protected]> 1460016747 -0700
|
||||
committer Reilly Tucker Siemens <[email protected]> 1460016747 -0700
|
||||
gpgsig -----BEGIN PGP SIGNATURE-----
|
||||
Version: GnuPG v1
|
||||
|
||||
iQIcBAABAgAGBQJXBhZrAAoJEBtFjnx8sVSpZkMP/jQk4lT+b0kpOj+VfvW3JREH
|
||||
R5ghCTJneZTlQzZcgtvN2ztQXG3Unn2A0YpaoCWC6gve0uv3W75JbJ6//Jtm/udq
|
||||
LP2iiZO8Pp7QdaEvGKL4c0nw/GgBYNicOcL61QWR1ymoK4d3FTGU3dEYMOOWN673
|
||||
vR4DVvv2DGD2OO3VAjpXmznJBGER5k5dtQ5asScWfYej2hEXQfESrWCT1BiXtqxA
|
||||
d/ge92C7t4zMFHs+LYdnXGoRYahQyCTfiIPaDQ9XdDREYMiA0dj6uahKWPhKzYnK
|
||||
89qXphF3PN1huJKN31eTANuiA2Pt3Swe/RYHOv+l8PcInFZWcmF7uQQ7Eivh+Hi7
|
||||
lO9l7XR9qIiW9r9890V25F2ESTSxoHMpcZDyV9lTDUYEBJgsP6v1C4JG2CvYVkkL
|
||||
vvqVb3CMldDdNvLnavFxmEmIPDNMDLrZR0s0yc5FdYBsADw0VG6QwG3j/IdSyOyH
|
||||
t4QzlFqGelm5vUBiqmZxOJE90LRI4e2876ZI5VPYmJ49mPpU4qNkRMQvVZLwtqOe
|
||||
mm616Ja5IEivM/1BKWIId9kTPB4/TzdgTRR6OJYwKcbdkiSdRIdhWbSe4c1VsTpd
|
||||
YQ6zpzg63/Mm4N3I/4pNXY3AGOa4JtSttRtTjeLnXyStZ47AwthcTB6ioIgLGJQ7
|
||||
8tEpiPvkr127Mj6VeMFv
|
||||
=AgDl
|
||||
-----END PGP SIGNATURE-----
|
||||
|
||||
Why doesn't this work?
|
||||
```
|
||||
|
||||
**Whoa.** While this may have already been obvious to some of you I had no idea
|
||||
that git objects had separate authors and committers. GitHub was right all
|
||||
along. The email in my signature _doesn't_ match the committer email. I'll bet
|
||||
I can leverage `git filter-branch` again to finally fix this.
|
||||
|
||||
## How to Actually Actually Fix It (A.K.A Tell Git Who's Boss)
|
||||
|
||||
Just as there is a `GIT_AUTHOR_EMAIL` environment variable to use in a filter,
|
||||
there is also a `GIT_COMMITTER_EMAIL`. Now I can simply
|
||||
|
||||
```bash
|
||||
git filter-branch -f --env-filter "GIT_AUTHOR_EMAIL='[email protected]'; GIT_COMMITTER_EMAIL='[email protected]'" --msg-filter 'sed "/iQIcBA.*/,/.*END PGP SIGNATURE.*/d"' --commit-filter 'git commit-tree -S "$@"' HEAD
|
||||
```
|
||||
|
||||
and _voila!_, `git cat-file -p fd23e7c` shows a commit object with the correct
|
||||
author **and** commiter.
|
||||
|
||||
```
|
||||
tree 8feaf4ea13ac445111d9213cd5f917085e381642
|
||||
parent 244b168ef6fc8fb5aef6abf4a68426299c00e0f1
|
||||
author Reilly Tucker Siemens <[email protected]> 1457219418 -0800
|
||||
committer Reilly Tucker Siemens <[email protected]> 1457219418 -0800
|
||||
gpgsig -----BEGIN PGP SIGNATURE-----
|
||||
Version: GnuPG v1
|
||||
|
||||
iQIcBAABAgAGBQJXBhuzAAoJEBtFjnx8sVSpXY4QAK5mkQbuAplY9e7FcwR5CB2Q
|
||||
7ZwgKBJccmkBxWaH6UTsrFJzMXBzpIXnkycUStGMD0tduNo/jxnK19QBPaDVuQ0C
|
||||
oD6RIIgTEsuJ81IserBwILryr6G7MBWQp5qWbXrCztN+SAXg7S3Rh235S6t64HtW
|
||||
FwZ8gBWz+tUhr9ysrOYEilXYDiltO5QRHrVbE0QBGV0FRVHgSlnUeChZaTiYT5+T
|
||||
pHHBezipNqMTnbiRGyc8/yfrfD32YljSRrZKH4ly4sNdUklJKUraoaIZNojybk2f
|
||||
DxZmXgvHlcfIJJO9WzL2KEoCpWMg8hQXM1CQf7u+98hBcWe/1J8E2Wt6mbo04r9J
|
||||
uPYyerLoIgKqAKJj4CZeFCPOzl3N9OPQHTN1aamq6/td5E8MRTxf2vxHP2GS/vOX
|
||||
yFNjBlxy63u1yBi6u79iKvdzjG933Z/MYONOAnSxHvk1Ka79lIh4G49Gk7AMUlON
|
||||
IxJ/PRzf8CjTzz6jaoZQIHG+BkEHXIiT2YZRSNKA4vYyL/iKj9OmXK/0SeSDeELh
|
||||
/4uByx37dgAYW9hZho3d2+BiW5pVsaDOUNLpStu+c4u31juZMlM5OkskvCpAI00a
|
||||
/1F8s1LraG9GxEkyeWNnAm9wV4JWkBjSvfUraj2jeHdotkIpJ4FEOSXpm+wHySX0
|
||||
wd1/IAJz3DesEmKlTIMW
|
||||
=LZcd
|
||||
-----END PGP SIGNATURE-----
|
||||
|
||||
|
||||
Add cookiecutter configuration.
|
||||
```
|
||||
|
||||
Unfortunately, now the graph of my `git log` looks weird. I have double the
|
||||
commits!
|
||||
|
||||
<figure>
|
||||
<img src='messed-up-git-graph.png' alt='Messed Up Git Graph'>
|
||||
<figcaption>
|
||||
Not <em>quite</em> there yet. <a href='messed-up-git-graph.png'>View full size.</a>
|
||||
</figcaption>
|
||||
</figure>
|
||||
|
||||
## Success
|
||||
|
||||
For the third time [Stack Overflow saves my bacon] and
|
||||
```bash
|
||||
git update-ref -d refs/original/refs/heads/master
|
||||
```
|
||||
|
||||
cleans up my graph. Now I can force push to `origin master` _again_ and
|
||||
everything will be right again.
|
||||
|
||||
<figure>
|
||||
<img src='verified-commits.png' alt='Verified Commits'>
|
||||
<figcaption>
|
||||
Verified commits on GitHub. Success! <a href='verified-commits.png'>View full size.</a>
|
||||
</figcaption>
|
||||
</figure>
|
||||
|
||||
[dotfiles]: https://github.com/reillysiemens/dotfiles
|
||||
[GPG signature verification]: https://github.com/blog/2144-gpg-signature-verification
|
||||
[public key]: https://github.com/reillysiemens.gpg
|
||||
[change the authorship]: https://stackoverflow.com/questions/750172/change-the-author-of-a-commit-in-git
|
||||
[using a signed tag]: https://stackoverflow.com/questions/13504983/retroactively-sign-git-commits
|
||||
[Additional Stack Overflowing]: http://stackoverflow.com/a/31640257/3288364
|
||||
[Git Immersion]: http://gitimmersion.com/lab_11.html
|
||||
[Stack Overflow saves my bacon]: http://stackoverflow.com/a/7654880/3288364
|
||||
|
Before Width: | Height: | Size: 226 KiB |
|
Before Width: | Height: | Size: 58 KiB |
|
Before Width: | Height: | Size: 41 KiB |
|
Before Width: | Height: | Size: 31 KiB |
@@ -1,112 +0,0 @@
|
||||
+++
|
||||
title = "gutenberg init blog"
|
||||
description = "Switching to Gutenberg as my static site generator."
|
||||
date = 2017-06-18T15:17:00-08:00
|
||||
[taxonomies]
|
||||
tags = ["gutenberg", "Rust", "blogging"]
|
||||
+++
|
||||
When I first created this site I wanted to get it live as quickly as possible.
|
||||
[Hexo][hexo], a blogging framework written in Node.js, seemed like the perfect
|
||||
tool. At the time I was rather interested in Node.js, so it seemed natural to
|
||||
use a framework rooted in that community.
|
||||
|
||||
By the time of my last post I'd become increasingly disinterested in Node.js
|
||||
and much more interested in Rust and its community. It was mostly
|
||||
procrastination, but I convinced myself that using a tool written in a language
|
||||
I didn't use often directly contributed to the paucity of posts here, so I
|
||||
finally decided to ditch Hexo.
|
||||
|
||||
<!-- more -->
|
||||
|
||||
## Replacement Criteria
|
||||
|
||||
Of course, I needed a suitable replacement. I wanted it to
|
||||
|
||||
- Be fast
|
||||
- Be flexible
|
||||
- Generate [RSS][RSS] feeds
|
||||
- Support live reloading
|
||||
- Support [Markdown][CommonMark]
|
||||
|
||||
Fast and flexible are nebulous requirements. Most static site generators are
|
||||
more than fast enough since I'm not generating a site with hundreds of pages
|
||||
(yet). My [early experience with Hexo][early Hexo] taught me that while having
|
||||
a bunch of features out of the box is nice flexibility is more important. I
|
||||
needed to be able to choose my own deployment mechanisms and structure my site
|
||||
the way I wanted to.
|
||||
|
||||
I don't personally use an RSS feed reader, but I know friends of mine do and
|
||||
they've asked me to support that ([here ya go][feed link]). I sure as heck
|
||||
didn't want to generate an RSS feed myself.
|
||||
|
||||
Live reloading and Markdown support were non-negotiable requirements. The
|
||||
write, build, and reload cycle is too tedious to forgo live reloading. I simply
|
||||
didn't want to use a markup language other than Markdown because writing it is
|
||||
effortless for me at this point.
|
||||
|
||||
## The Hunt
|
||||
|
||||
I [surveyed the landscape][staticgen], but didn't find
|
||||
anything to my liking, so I procrastinated even more. Then I happened across an
|
||||
[article][Introducing Tera] linked from an [orange website][HN Tera post]
|
||||
introducing [Tera][Tera], a template engine in Rust. When I saw that the author
|
||||
drew inspiration from Python's [Jinja2][Jinja2] templating library I got
|
||||
_really_ excited.
|
||||
|
||||
I tried my hand at writing my own static site generator in Rust. I managed to
|
||||
get a rudimentary POC using Tera operational just in time for
|
||||
[Vincent Prouillet][Vincent], the author of Tera, to
|
||||
[announce Gutenberg][announcing Gutenberg]. It met all my replacement criteria,
|
||||
used the templating engine I was interested in, and was written in Rust!
|
||||
|
||||
## Building My New Site
|
||||
|
||||
Once I found Gutenberg I attacked fixing up my site with [vigor][vigor]. I
|
||||
wrote my own templates, learned a little [Sass][Sass], and even managed to make
|
||||
a [small contribution to Gutenberg][Gutenberg Contribution]. Hexo made my site
|
||||
largely a black box, but now there isn't a line of the source I haven't
|
||||
touched.
|
||||
|
||||
As an added bonus, I took the redesign opportunity to ensure there's **no
|
||||
JavaScript anywhere on this site**, so this page renders exactly the same
|
||||
regardless of whether you're using a JavaScript blocker. In the future I may
|
||||
relax this constraint, but I'm happy with my decision so far and I was able to
|
||||
learn more about Flexbox and CSS3 as a result.
|
||||
|
||||
I also had time to add custom [**404 Not Found**][404] and
|
||||
[**50x Server Error**][50x] pages. If you're interested in more little details,
|
||||
you can find the source for the redesigned site on [GitHub][GitHub repo].
|
||||
|
||||
## Credit Where Credit Is Due
|
||||
|
||||
This would not have been possible without Vincent's hard work on Tera and
|
||||
Gutenberg. I also borrowed a great deal from [Alex Sun][Alex Sun]'s
|
||||
[vida][vida] Jekyll theme in writing this site's Sass. I've learned a lot from
|
||||
both of them. Thanks!
|
||||
|
||||
## What's Next?
|
||||
|
||||
This site is in a much better place than it was a year ago. I understand it
|
||||
better and I'm more motivated to continue working on it, so expect more posts!
|
||||
|
||||
|
||||
[hexo]: https://hexo.io/
|
||||
[RSS]: https://en.wikipedia.org/wiki/RSS
|
||||
[CommonMark]: http://commonmark.org/
|
||||
[early Hexo]: /posts/hexo-init-blog
|
||||
[feed link]: /rss.xml
|
||||
[staticgen]: https://www.staticgen.com/
|
||||
[Introducing Tera]: https://www.vincentprouillet.com/blog/introducing-tera/
|
||||
[HN Tera post]: https://news.ycombinator.com/item?id=11507188
|
||||
[Tera]: https://github.com/Keats/tera
|
||||
[Jinja2]: http://jinja.pocoo.org/
|
||||
[Vincent]: https://www.vincentprouillet.com/
|
||||
[announcing Gutenberg]: https://www.vincentprouillet.com/blog/announcing-gutenberg/
|
||||
[vigor]: https://github.com/reillysiemens/tuckersiemens.com/graphs/contributors?from=2017-04-23&to=2017-06-14&type=c
|
||||
[Sass]: http://sass-lang.com/
|
||||
[Gutenberg Contribution]: https://github.com/Keats/gutenberg/commit/7c1cecb2112e087e2ffb45937bea29ebf81df01e
|
||||
[404]: /404.html
|
||||
[50x]: /50x.html
|
||||
[GitHub repo]: https://github.com/reillysiemens/tuckersiemens.com
|
||||
[Alex Sun]: https://syaning.github.io/
|
||||
[vida]: https://github.com/syaning/vida
|
||||
@@ -0,0 +1,36 @@
|
||||
+++
|
||||
title = "Hello World"
|
||||
description = "Hello world"
|
||||
date = 2022-11-04
|
||||
author = "Ian Andwati"
|
||||
[taxonomies]
|
||||
tags = ["hugo", "blog", "go"]
|
||||
+++
|
||||
|
||||
How it begins
|
||||
|
||||
<!-- more -->
|
||||
|
||||
In this post-web2.0 age, people seem to blog less and less, and blogs seem to be dying slowly. People are separated in different Apps where the content is not searchable by search engines.
|
||||
|
||||
However, I still prefer to write using a blog.
|
||||
|
||||
# Why this?
|
||||
|
||||
The best way to learn something is to teach others the knowledge that you have gained. Regurgitating that information will solidify it more firmly in your mind.
|
||||
|
||||
# What I write?
|
||||
|
||||
Mostly random stuff and a few tutorials. I'll see where this goes.Although some of the blog posts may only be helpful to myself, but who knows, I hope they will be helpful to someone else in some day.
|
||||
|
||||
# Static blog generator
|
||||
|
||||
I use hugo as a static site generator.After trying a couple of blogging platforms and trying to build a few from scratch I decided to
|
||||
throw in the towel and opt in for a ready made solution, no need reinventing the wheel.
|
||||
|
||||
Hugo is one of the most popular open-source static site generators. With its amazing speed and flexibility, Hugo makes building websites fun again.Hugo provides a robust theming system that is easy to implement but capable of producing even the most complicated websites.Hugo's Go-based templates provide just the right amount of logic to build anything from the simple to complex. For this site I decided to go with the [even theme](https://github.com/olOwOlo/hugo-theme-even). The commenting system is implemented using [Utterances](https://utteranc.es/)
|
||||
|
||||
# Blog site hosting
|
||||
|
||||
The site is hosted on github pages as evident by the \*.github.io domain name.I prefer this mainly because it is free and I don't have
|
||||
to switch to a different hosting provider.
|
||||
@@ -1,66 +0,0 @@
|
||||
+++
|
||||
title = "hexo init blog"
|
||||
description = "Hello World!"
|
||||
date = 2015-04-24T01:57:13-08:00
|
||||
[taxonomies]
|
||||
tags = ["hacks", "Hexo", "blogging"]
|
||||
+++
|
||||
I've been wanting to start a blog for a long time now. Today I'm pulling the
|
||||
trigger on that with a simple `hexo init blog`. Well, it wasn't _that_ simple,
|
||||
so I feel like it's worth talking about a few of the complications I had.
|
||||
|
||||
<!-- more -->
|
||||
|
||||
|
||||
[Hexo](https://hexo.io/), my chosen blogging framework, has made some
|
||||
interesting decisions. Running `tree -aACL 1` shows the directory structure for
|
||||
my new Hexo blog.
|
||||
|
||||
```
|
||||
.
|
||||
├── _config.yml
|
||||
├── db.json
|
||||
├── .deploy_git
|
||||
├── .gitignore
|
||||
├── node_modules
|
||||
├── package.json
|
||||
├── public
|
||||
├── scaffolds
|
||||
├── source
|
||||
└── themes
|
||||
|
||||
6 directories, 4 files
|
||||
```
|
||||
|
||||
Hexo keeps its generated static files in the `public` directory. When deploying
|
||||
with git, as I choose to do, it uses the hidden `.deploy_git` directory to keep
|
||||
changes to `public` under version control. When generating static files Hexo
|
||||
obliterates anything in the `public` directory it doesn't know about. This
|
||||
means that even if the user wants to add some static files of their own they'll
|
||||
just be [baleeted](https://www.youtube.com/watch?v=07h0ksKx5sM) when Hexo
|
||||
clears out the `.deploy_git` directory and copies the `public` directory in.
|
||||
|
||||
|
||||
This isn't a big deal if you don't care about having any static files not
|
||||
generated by Hexo on your site. It just so happens that I'm very keen on having
|
||||
my [proof](https://tuckersiemens.com/keybase.txt) for
|
||||
[Keybase](https://keybase.io). In order to prove ownership of a website Keybase
|
||||
requires that you keep a file called `keybase.txt` or `.well-known/keybase.txt`
|
||||
from the root of your site.
|
||||
|
||||
|
||||
So, how do you serve a custom static file when your blogging framework is
|
||||
intent on keeping you from doing so? I suppose I could do the responsible thing
|
||||
and submit a pull request upstream for a configuration option to preserve
|
||||
certain files, but I'm much too lazy for that. Instead I've enlisted
|
||||
[nginx](http://nginx.org/) for a band-aid solution.
|
||||
|
||||
```
|
||||
location =/keybase.txt {
|
||||
root /srv;
|
||||
}
|
||||
```
|
||||
|
||||
Most requests to my site pass through nginx to my default root, but any
|
||||
requests for `keybase.txt` get served out of a separate directory where it
|
||||
safely sits.
|
||||
@@ -0,0 +1,172 @@
|
||||
+++
|
||||
title ="Install PostgreSQL on Arch | Manjaro | Garuda Linux"
|
||||
description = "Getting PostgreSQL up and running on Arch Linux-based distros"
|
||||
date = 2023-02-16T15:34:14.160Z
|
||||
author = "Ian Andwati"
|
||||
[taxonomies]
|
||||
tags = ["postgresql", "archlinux", "manjaro", "garuda", "linux"]
|
||||
+++
|
||||
|
||||
# Overview
|
||||
|
||||
**PostgreSQL** is a powerful open-source relational database management system (RDBMS) that allows you to store and manage large amounts of data. It was first released in 1996 and has become one of the most popular RDBMSs in the world.
|
||||
|
||||
PostgreSQL is designed to handle high levels of **concurrency**, providing features such as **MVCC (multi-version concurrency control)** that allow multiple users to access the same data simultaneously without conflicts. It is also **highly extensible**, with a large number of third-party extensions and libraries available that can be used to add functionality to the core database.
|
||||
|
||||
Some of the key features of PostgreSQL include support for **complex SQL queries**, **advanced indexing** and **query optimization**, **transactional integrity**, and support for a **wide range of programming languages** and development frameworks.PostgreSQL is used by a wide range of organizations, from small businesses to large enterprises, and is particularly popular in industries such as finance, healthcare, and e-commerce, where high levels of data security and reliability are critical.
|
||||
|
||||
In this tutorial, we’ll look at how to install PostgreSQL on Arch Linux, Manjaro, Garuda Linux, or any other Arch-based distro out there.
|
||||
|
||||
# PostgreSQL use cases
|
||||
|
||||
- **E-commerce websites:** PostgreSQL is a popular choice for e-commerce websites because it can handle large amounts of data and supports complex queries. It is used for product catalogs, order management, customer data, and more.
|
||||
- **Financial institutions:** PostgreSQL is used in financial institutions to store and manage transactional data, such as stock trades, bank transfers, and credit card transactions. It is highly secure and provides features such as transactional integrity and data encryption.
|
||||
- **Healthcare industry:** PostgreSQL is used in the healthcare industry to store patient data, such as medical records and diagnostic test results. It is also used for research purposes, where large amounts of data need to be stored and analyzed.
|
||||
- **Government agencies:** PostgreSQL is used by government agencies for a variety of purposes, including managing large datasets, storing and analyzing census data, and managing public health information.
|
||||
- **Mobile and web applications:** PostgreSQL is often used as a back-end database for mobile and web applications. It provides reliable data storage and retrieval and supports a wide range of programming languages and development frameworks.
|
||||
- **Geospatial applications:** PostgreSQL has built-in support for geospatial data and is often used for applications that require location-based data, such as mapping and GPS tracking.
|
||||
|
||||
# Why PostgreSQL?
|
||||
|
||||
Postgres offers a lot of advantages that make it an excellent choice for highly transactional environments regardless of application size and data volume. Its rich feature set and flexibility make it a popular choice for a wide range of applications and industries. Some of the top features are:
|
||||
|
||||
- **ACID Compliance:** PostgreSQL is ACID (Atomicity, Consistency, Isolation, Durability) compliant, which means it provides high reliability, consistency, and data integrity, even in the presence of hardware failures, power outages, and other system errors.
|
||||
- **Extensibility:** PostgreSQL is highly extensible, allowing developers to add new data types, operators, and functions using its extension framework. This makes it easy to customize PostgreSQL to meet the specific needs of your application.
|
||||
- **Concurrency:** PostgreSQL provides a high level of concurrency, allowing multiple users to access the same data at the same time without conflicts. It uses a Multi-Version Concurrency Control (MVCC) model that enables fast and reliable access to data, even under heavy load.
|
||||
- **Full-text search:** PostgreSQL provides a powerful full-text search engine that supports advanced search capabilities, such as stemming, ranking, and relevance matching.
|
||||
- **Advanced indexing:** PostgreSQL provides a wide range of indexing options, including B-tree, Hash, GiST, SP-GiST, GIN, and BRIN. These indexing options allow developers to optimize database performance for their specific use cases.
|
||||
- **JSON support:** PostgreSQL provides native support for storing and querying JSON data, making it easy to work with modern web and mobile applications that rely on JSON for data exchange.
|
||||
- **Security:** PostgreSQL provides a high level of security, with features such as SSL support, client authentication, and role-based access control. It also provides encryption for both data at rest and data in transit.
|
||||
- **Scalability:** PostgreSQL is designed to scale horizontally and vertically, allowing you to add more nodes to a cluster or increase the resources of a single node to handle more load.
|
||||
|
||||
# Install PostgreSQL on Arch | Manjaro | Garuda Linux
|
||||
|
||||
Now with all that fluff out of the way, let's get to business. The steps have been tested on Arch Linux with the KDE desktop.
|
||||
|
||||
> **Note:** Commands that should be run as the *postgres* user are prefixed by `[postgres]$` in this article.
|
||||
|
||||
## Step 1: Installing the PostgreSQL Package
|
||||
|
||||
```sh
|
||||
sudo pacman -Syu postgresql
|
||||
```
|
||||
|
||||
This command will update the package database and update all packages on the system then install the PostgreSQL package from your distribution mirrors. Never install a package without updating the system first. On a rolling release, this can lead to an unbootable system. Installing the package will also create a system user called *postgres*. You can now switch to the *postgres* user using a privilege elevation program.
|
||||
|
||||
You can switch to the PostgreSQL user by executing the following command:
|
||||
|
||||
- If you have sudo and are in sudoers
|
||||
|
||||
```sh
|
||||
sudo -iu postgres
|
||||
```
|
||||
|
||||
- Otherwise using su:
|
||||
|
||||
```sh
|
||||
su
|
||||
su -l postgres
|
||||
```
|
||||
|
||||
## Step 2: Initial Configuration
|
||||
|
||||
Before PostgreSQL can function correctly, the database cluster must be initialized.
|
||||
|
||||
### 2.1 We can confirm the installed PostgreSQL version by running:
|
||||
|
||||
```sh
|
||||
postgres --version ─╯
|
||||
postgres (PostgreSQL) 15.1
|
||||
```
|
||||
|
||||
### 2.2 Set applicable entries in **/etc/locale.gen**
|
||||
|
||||
```sh
|
||||
echo "en_US.UTF-8 UTF-8" | sudo tee /etc/locale.gen
|
||||
```
|
||||
|
||||
### 2.3 Then run **\*\*locale-gen\*\*** to generate locale settings
|
||||
|
||||
```sh
|
||||
sudo locale-gen
|
||||
Generating locales...
|
||||
en_US.UTF-8... done
|
||||
Generation complete.
|
||||
```
|
||||
|
||||
### 2.4 Initialize Postgres Data Directory
|
||||
|
||||
You must first log in as the **postgres** user using the following command before you can initialize PostgreSQL’s data directory:
|
||||
|
||||
```sh
|
||||
sudo su - postgres
|
||||
```
|
||||
|
||||
### 2.5 With the following command, you can now initialize PostgreSQL’s data directory:
|
||||
|
||||
```sh
|
||||
[postgres]$ initdb -D /var/lib/postgres/data
|
||||
```
|
||||
|
||||
### Step 3: PostgreSQL Service Management
|
||||
|
||||
### 3.1 Check PostgreSQL status with the following command:
|
||||
|
||||
```sh
|
||||
systemctl status postgresql ─╯
|
||||
postgresql.service - PostgreSQL database server
|
||||
Loaded: loaded (/usr/lib/systemd/system/postgresql.service; disabled; preset: disabled)
|
||||
Active: inactive (dead)
|
||||
```
|
||||
|
||||
### 3.2 Start the PostgreSQL service by running the following command:
|
||||
|
||||
```sh
|
||||
sudo systemctl start postgresql
|
||||
```
|
||||
|
||||
### 3.3 Enable the PostgreSQL service to start automatically at boot by running the following command:
|
||||
|
||||
```sh
|
||||
sudo systemctl enable postgresql
|
||||
```
|
||||
|
||||
### 3.4 Verify that PostgreSQL is running by running the following command:
|
||||
|
||||
```sh
|
||||
sudo systemctl status postgresql ─╯
|
||||
● postgresql.service - PostgreSQL database server
|
||||
Loaded: loaded (/usr/lib/systemd/system/postgresql.service; enabled; preset: disabled)
|
||||
Active: active (running) since Thu 2023-02-16 22:52:43 EAT; 20s ago
|
||||
Main PID: 26106 (postgres)
|
||||
Tasks: 6 (limit: 8790)
|
||||
Memory: 16.9M
|
||||
CPU: 113ms
|
||||
CGroup: /system.slice/postgresql.service
|
||||
├─26106 /usr/bin/postgres -D /var/lib/postgres/data
|
||||
├─26107 "postgres: checkpointer "
|
||||
├─26108 "postgres: background writer "
|
||||
├─26110 "postgres: walwriter "
|
||||
├─26111 "postgres: autovacuum launcher "
|
||||
└─26112 "postgres: logical replication launcher "
|
||||
```
|
||||
|
||||
### Change postgres user password
|
||||
|
||||
To change the postgres user password execute the following command:
|
||||
|
||||
```sh
|
||||
sudo -u postgres psql -c "ALTER USER postgres PASSWORD '$PGPASSWORD';"
|
||||
```
|
||||
|
||||
# Resources
|
||||
|
||||
- [Arch Linux Wiki](https://wiki.archlinux.org/title/PostgreSQL)
|
||||
- [Wikipedia](https://en.wikipedia.org/wiki/PostgreSQL)
|
||||
- [PostgreSQL Documentation](https://www.postgresql.org/docs/)
|
||||
|
||||
# Conclusion
|
||||
|
||||
Installing PostgreSQL on Arch Linux-based distribution is a straightforward process that can be completed in just a few steps. By following the steps outlined above, you can install, configure, and start the PostgreSQL database server on your Arch Linux system. Once installed, you can use PostgreSQL to store, manage, and query data for a wide range of applications and use cases. With its advanced features and high level of extensibility, PostgreSQL is a popular choice for developers and businesses seeking a robust and reliable database management system.
|
||||
|
||||
You can connect to the PostgreSQL server using the psql command-line client or a graphical tool such as pgAdmin.
|
||||
@@ -0,0 +1,159 @@
|
||||
+++
|
||||
title = "Integrating Netlify CMS"
|
||||
description = "Integrating netlify cms"
|
||||
date = 2022-11-24T15:34:14.160Z
|
||||
author = "Ian Andwati"
|
||||
[taxonomies]
|
||||
tags = ["blog", "hugo", "netlify","netlify-cms","github"]
|
||||
+++
|
||||
|
||||
Migrating to a better content management system through Netlify CMS
|
||||
|
||||
<!-- more -->
|
||||
|
||||
# Heavy Lifting
|
||||
|
||||
I use [Visual Studio Code](https://code.visualstudio.com/) to manually edit the blog post content on this website. This is quite a hassle because I have to deal with
|
||||
markdown files. There are many problems with this approach and several gotchas. I have to work on a local site repository to be able to edit the content. My GitHub credentials are also required to push my local changes to my remote repository on GitHub so that it can trigger the build process and publish the site.
|
||||
|
||||
This process has helped me a lot to master markdown syntax and some practice on my git skills. It’s fun to update the website content following these steps but sometimes it can be frustrating. At those times, I really miss the user interface of content management systems. After careful consideration, I moved on with Netlify CMS. At its core, Netlify CMS is an open-source React app that acts as a wrapper for the Git workflow, using the GitHub, GitLab, or Bitbucket API. This provides many advantages, including:
|
||||
|
||||
- **Fast, web-based UI:** With rich-text editing, real-time preview, and drag-and-drop media uploads.
|
||||
- **Platform agnostic:** Works with most static site generators.
|
||||
- **Easy installation:** Add two files to your site and hook up the backend by including those files in your build process or linking to our Content Delivery Network (CDN).
|
||||
- **Modern authentication:** Using GitHub, GitLab, or Bitbucket and JSON web tokens.
|
||||
- **Flexible content types:** Specify an unlimited number of content types with custom fields.
|
||||
- **Fully extensible:** Create custom-styled previews, UI widgets, and editor plugins.
|
||||
|
||||
# Redeploying to Netlify
|
||||
|
||||
I decided to move the site from GitHub pages to Netlify hosting to take full control of the platform. Hugo sites on Netlify can benefit from automatic framework detection and control over Hugo version selection. When you link a repository for a project, Netlify tries to detect the framework your site is using. If your site is built with Hugo, Netlify provides a suggested build command and publish directory: Hugo and public. The build process is straight forward but you can add custom options to your `netlify.toml` file, here are my options:
|
||||
|
||||
```toml
|
||||
[build]
|
||||
command = "hugo"
|
||||
publish = "public"
|
||||
|
||||
[build.environment]
|
||||
HUGO_VERSION = "0.105.0"
|
||||
```
|
||||
|
||||
# Setting up Netlify CMS
|
||||
|
||||
You can adapt Netlify CMS to a wide variety of projects. It works with any content written in markdown, JSON, YAML, or TOML files, stored in a repo on GitHub, GitLab, or Bitbucket. You can Netlify CMS to a site that's built with a common static site generator, like Jekyll, Hugo, Hexo, or Gatsby. The official documentation for installing on your website can be found at the [official site](https://www.netlifycms.org/docs/intro/)
|
||||
|
||||
## Steps
|
||||
|
||||
**Step 1**
|
||||
|
||||
Create a static admin folder containing all Netlify CMS files, stored at the root of your published site. Where you store this folder in the source files depends on your static site generator. For Hugo, it is the `static` folder.
|
||||
|
||||
`/static/admin`
|
||||
|
||||
**Step 2**
|
||||
|
||||
Inside the admin folder, create two files: `index.html` and `config.yml`. The first file `admin/index.html` is an entry point to Netlify CMS. You will access the CMS interface from `yourwebsite.com/admin` when everything is ready.
|
||||
|
||||
**Step 3**
|
||||
|
||||
The first file, `admin/index.html`, is the entry point for the Netlify CMS admin interface. Paste this code in index.html and save. This HTML starter page will load the Netlify CMS Javascript file:
|
||||
|
||||
```html
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<title>Content Manager</title>
|
||||
</head>
|
||||
<body>
|
||||
<!-- Include the script that builds the page and powers Netlify CMS -->
|
||||
<script src="https://unpkg.com/netlify-cms@^2.0.0/dist/netlify-cms.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
```
|
||||
|
||||
**Step 4**
|
||||
|
||||
Now let’s move to the configuration section. These are the configurations that go into the `config.yml` file. This process is different for every site. I followed the guidelines on the website and made minor adjustments. It worked on my website. The configuration options are available at the [docs](<https://www.netlifycms.org/docs/add-to-your-site/#:~:text=%2C%20MyTemplate)-,Configuration,-Configuration%20is%20different>). This is how I configured my config.yml file:
|
||||
|
||||
```yaml
|
||||
backend:
|
||||
name: git-gateway
|
||||
branch: netlify-cms # Branch to update (optional; defaults to master)
|
||||
|
||||
# This line should *not* be indented
|
||||
publish_mode: editorial_workflow
|
||||
|
||||
media_folder: "static/images/uploads" # Media files will be stored in the repo under static/images/uploads
|
||||
public_folder: "/images/uploads" # The src attribute for uploaded media will begin with /images/uploads
|
||||
|
||||
collections:
|
||||
- name: "post" # Used in routes, e.g., /admin/collections/blog
|
||||
label: "posts" # Used in the UI
|
||||
folder: "content/post" # The path to the folder where the documents are stored
|
||||
create: true # Allow users to create new documents in this collection
|
||||
slug: "{{year}}-{{month}}-{{day}}-{{slug}}" # Filename template, e.g., YYYY-MM-DD-title.md
|
||||
fields: # The fields for each document, usually in front matter
|
||||
- { label: "Layout", name: "layout", widget: "hidden", default: "blog" }
|
||||
- { label: "Title", name: "title", widget: "string" }
|
||||
- { label: "Draft", name: "draft", widget: "boolean", default: true }
|
||||
- { label: "Publish Date", name: "date", widget: "datetime" }
|
||||
- { label: "Last Modified", name: "lastmod", widget: "datetime" }
|
||||
- { label: "Body", name: "body", widget: "markdown" }
|
||||
- { label: "Keywords", name: "keywords", widget: "list" }
|
||||
- { label: "Tags", name: "tags", widget: "list" }
|
||||
- { label: "Categories", name: "categories", widget: "list" }
|
||||
- { label: "Comments", name: "comment", widget: "boolean", default: true }
|
||||
- {
|
||||
label: "Table of Contents",
|
||||
name: "toc",
|
||||
widget: "boolean",
|
||||
default: true,
|
||||
}
|
||||
```
|
||||
|
||||
**Step 5**
|
||||
|
||||
**Authentication**
|
||||
|
||||
Now that the Netlify CMS files are in place and configured, all that's left is to enable authentication.
|
||||
Netlify offers a built-in authentication service called Identity. In order to use it, connect your site repo with Netlify. Netlify has published a general [Step-by-Step Guide](https://www.netlify.com/blog/2016/10/27/a-step-by-step-guide-deploying-a-static-site-or-single-page-app/) for this, along with detailed guides for many popular static site generators
|
||||
|
||||
Let’s first enable Netlify Identity and Git Gateway:
|
||||
|
||||
- On your Netlify account, go to **Settings > Identity** and select **Enable Identity Service**
|
||||
- Under **Registration preferences**, select **Invite Only**.
|
||||
- If you want to enable login from external providers such as Google and GitHub, check the boxes you want to use, under **External Providers** section. I find it practical to log in with GitHub, so I added it.
|
||||
- Go to **Services > Git Gateway**, and click **Enable Git Gateway**. This authenticates with your Git Host and generates and API token.
|
||||
- Everything is almost set up. What we need now is a front-end interface to connect. We will add the following script in two places, to include the **Netlify Identity Widget** on our website:
|
||||
|
||||
```html
|
||||
<script src="https://identity.netlify.com/v1/netlify-identity-widget.js"></script>
|
||||
```
|
||||
|
||||
We will use Netlify's [script injection](https://docs.netlify.com/site-deploys/post-processing/snippet-injection/) feature to add the script to our site. To use snippet injection, go to **Site Settings > Build & Deploy > Post processing**. Find the Snippet Injection section and select **Add Snippet**. Write a name for the script in the first box (e.g. Netlify Identity Widget). Paste the script inside the second box. Choose Insert before the `</body>` option and save.
|
||||
|
||||
Lastly, we will add the following script before the closing `<body>` tag of our website’s main index page. This script allows the user to redirect back to the `/admin/` path after completing the login with the Netlify Identity Widget:
|
||||
|
||||
```html
|
||||
<script>
|
||||
if (window.netlifyIdentity) {
|
||||
window.netlifyIdentity.on("init", (user) => {
|
||||
if (!user) {
|
||||
window.netlifyIdentity.on("login", () => {
|
||||
document.location.href = "/admin/";
|
||||
});
|
||||
}
|
||||
});
|
||||
}
|
||||
</script>
|
||||
```
|
||||
|
||||
**Step 5**
|
||||
|
||||
Now we’re all set. Since we set our registration preferences to **“Invite Only”**, we need to invite ourselves as site users. To do this, select the Identity tab from the Netlify site dashboard and select the Invite Users button. Add your email address and send an invitation. You will receive an email. In your email, click **“accept the invitation”**. You will be directed to your website admin sign-up panel. Choose a password. Now you are ready to log in to Netlify CMS.
|
||||
|
||||
# Setting up a custom domain
|
||||
|
||||
I bought a custom domain from [Namecheap](https://namecheap.com), they provide some pretty good deals. After pointing the nameservers for the domain to the Netlify nameservers, voila! everything worked smoothly. For HTTPS support, Netlify enables automatic TLS certificates with Let’s Encrypt.
|
||||
@@ -0,0 +1,188 @@
|
||||
+++
|
||||
title = "Export default and module.exports in Javascript"
|
||||
description = "Modules are the building blocks of modern JavaScript applications, allowing for the encapsulation of code into reusable components. Two primary systems govern the exportation of these modules: export default in ES6 and module.exports in CommonJS. "
|
||||
author = "Ian Andwati"
|
||||
date = 2024-01-17
|
||||
[taxonomies]
|
||||
tags = ["javascript", "node"]
|
||||
+++
|
||||
|
||||
Javascript is a popular programming language that can run both in the browser and on the server. However, different environments may have different ways of organizing and importing Javascript modules. In this article, we will compare two common methods of exporting and importing modules: **export default** and **module.exports**.
|
||||
|
||||
|
||||
## What are modules?
|
||||
Modules are reusable pieces of code that can be separated into different files. Modules can export values, functions, classes, or objects that can be imported by other modules. This way, modules can avoid polluting the global scope and enable code reuse and modularity.
|
||||
|
||||
## How to export modules in Node.js?
|
||||
Node.js is a runtime environment that allows Javascript to run on the server. Node.js uses a module system called CommonJS, which is based on the require and module.exports syntax.
|
||||
|
||||
To export a module in Node.js, you can assign the value that you want to export to the module.exports object. For example:
|
||||
|
||||
```javascript
|
||||
// my-module.js
|
||||
// export a function
|
||||
module.exports = function add(a, b) {
|
||||
return a + b;
|
||||
};
|
||||
|
||||
// export an object
|
||||
module.exports = {
|
||||
name: "Alice",
|
||||
age: 25,
|
||||
};
|
||||
```
|
||||
|
||||
To import a module in Node.js, you can use the require function and pass the path of the module file as an argument. The require function will return the value of module.exports from the module file. For example:
|
||||
|
||||
```javascript
|
||||
// main.js
|
||||
// import a function
|
||||
const add = require("./my-module.js");
|
||||
console.log(add(2, 3)); // 5
|
||||
|
||||
// import an object
|
||||
const person = require("./my-module.js");
|
||||
console.log(person.name); // Alice
|
||||
console.log(person.age); // 25
|
||||
```
|
||||
|
||||
Note that Node.js will cache the modules that are imported with require, so if you import the same module multiple times, you will get the same instance of the module.
|
||||
|
||||
## How to export modules in ES6?
|
||||
ES6 is the version of Javascript that introduced many new features, including a native module system. ES6 modules use the `import` and `export` syntax.
|
||||
|
||||
To export a module in ES6, you can use the `export` keyword followed by the declaration of the value that you want to export. You can also use the export keyword with curly braces to export a list of named values. For example:
|
||||
|
||||
```javascript
|
||||
// my-module.js
|
||||
// export a function
|
||||
export function add(a, b) {
|
||||
return a + b;
|
||||
}
|
||||
|
||||
// export a class
|
||||
export class Person {
|
||||
constructor(name, age) {
|
||||
this.name = name;
|
||||
this.age = age;
|
||||
}
|
||||
}
|
||||
|
||||
// export an object
|
||||
export const person = {
|
||||
name: "Alice",
|
||||
age: 25,
|
||||
};
|
||||
```
|
||||
|
||||
To import a module in ES6, you can use the `import` keyword followed by the name of the value that you want to import and the path of the module file. You can also use the import keyword with curly braces to import a list of named values. For example:
|
||||
|
||||
```javascript
|
||||
// main.js
|
||||
// import a function
|
||||
import { add } from "./my-module.js";
|
||||
console.log(add(2, 3)); // 5
|
||||
|
||||
// import a class
|
||||
import { Person } from "./my-module.js";
|
||||
const person = new Person("Bob", 30);
|
||||
console.log(person.name); // Bob
|
||||
console.log(person.age); // 30
|
||||
|
||||
// import an object
|
||||
import { person as Alice } from "./my-module.js";
|
||||
console.log(Alice.name); // Alice
|
||||
console.log(Alice.age); // 25
|
||||
```
|
||||
Note that ES6 modules are imported and executed only once, so if you import the same module multiple times, you will get the same instance of the module.
|
||||
|
||||
## What is the difference between export default and module.exports?
|
||||
One of the main differences between export default and module.exports is that export default is an ES6 feature, while module.exports is a CommonJS feature. Therefore, they are not compatible with each other by default. However, some tools like Babel or Webpack can transpile ES6 modules to CommonJS modules, or vice versa, to make them work together.
|
||||
|
||||
Another difference is that export default can only export one value per module, while module.exports can export any value, including multiple values. For example:
|
||||
|
||||
```javascript
|
||||
// my-module.js
|
||||
// export default a function
|
||||
export default function add(a, b) {
|
||||
return a + b;
|
||||
}
|
||||
|
||||
// export default an object
|
||||
export default {
|
||||
name: "Alice",
|
||||
age: 25,
|
||||
};
|
||||
|
||||
// export default a class
|
||||
export default class Person {
|
||||
constructor(name, age) {
|
||||
this.name = name;
|
||||
this.age = age;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```javascript
|
||||
// my-module.js
|
||||
// export a function
|
||||
module.exports = function add(a, b) {
|
||||
return a + b;
|
||||
};
|
||||
|
||||
// export an object
|
||||
module.exports = {
|
||||
name: "Alice",
|
||||
age: 25,
|
||||
};
|
||||
|
||||
// export a class
|
||||
module.exports = class Person {
|
||||
constructor(name, age) {
|
||||
this.name = name;
|
||||
this.age = age;
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
Note that you can only have one export default statement per module, but you can have multiple module.exports statements. However, each module.exports statement will overwrite the previous one, so only the last one will be exported.
|
||||
|
||||
To import a default export, you can use the import keyword without curly braces and assign any name to the imported value. For example:
|
||||
|
||||
```javascript
|
||||
// main.js
|
||||
// import a default export
|
||||
import add from "./my-module.js";
|
||||
console.log(add(2, 3)); // 5
|
||||
|
||||
// import another default export
|
||||
import person from "./my-module.js";
|
||||
console.log(person.name); // Alice
|
||||
console.log(person.age); // 25
|
||||
|
||||
// import another default export
|
||||
import Person from "./my-module.js";
|
||||
const person = new Person("Bob", 30);
|
||||
console.log(person.name); // Bob
|
||||
console.log(person.age); // 30
|
||||
```
|
||||
|
||||
To import a module.exports value, you can use the require function and assign any name to the imported value. For example:
|
||||
```javascript
|
||||
// main.js
|
||||
// import a module.exports value
|
||||
const add = require("./my-module.js");
|
||||
console.log(add(2, 3)); // 5
|
||||
|
||||
// import another module.exports value
|
||||
const person = require("./my-module.js");
|
||||
console.log(person.name); // Alice
|
||||
console.log(person.age); // 25
|
||||
|
||||
// import another module.exports value
|
||||
const Person = require("./my-module.js");
|
||||
const person = new Person("Bob", 30);
|
||||
console.log(person.name); // Bob
|
||||
console.log(person.age); // 30
|
||||
```
|
||||
Node’s JavaScript has diverged from that of the browser. Understanding both CommonJS modules and ES6 modules is your first step in becoming a 10X developer. Happy coding!
|
||||
|
Before Width: | Height: | Size: 845 KiB |
@@ -1,608 +0,0 @@
|
||||
+++
|
||||
title = "node.example.com Is An IP Address"
|
||||
description = "node.example.com is an IP address."
|
||||
url = "posts/node-example-com-is-an-ip-address"
|
||||
date = 2020-12-28T06:13:05-08:00
|
||||
[taxonomies]
|
||||
tags = ["Python", "Testing", "Networking"]
|
||||
+++
|
||||
Hello! Welcome to the once-yearly blog post! This year I'd like to examine the
|
||||
most peculiar bug I encountered at work. To set the stage, let's start with a
|
||||
little background.
|
||||
|
||||
<!-- more -->
|
||||
|
||||
When we write [URLs][URL] with a [non-standard][standard ports] [port] we
|
||||
specify the port after a `:`. With [hostnames][hostname] and [IPv4] addresses
|
||||
this is straightforward. Here's some [Python] code to show how easy it is.
|
||||
|
||||
```python
|
||||
>>> url = urllib.parse.urlparse("https://node.example.com:8000")
|
||||
>>> (url.hostname, url.port)
|
||||
('node.example.com', 8000)
|
||||
>>>
|
||||
>>> url = urllib.parse.urlparse("https://192.168.0.1:8000")
|
||||
>>> (url.hostname, url.port)
|
||||
('192.168.0.1', 8000)
|
||||
```
|
||||
|
||||
Unfortunately, when [IPv6] addresses are involved some ambiguity is introduced.
|
||||
|
||||
```python
|
||||
>>> url = urllib.parse.urlparse(
|
||||
... "https://fdc8:bf8b:e62c:abcd:1111:2222:3333:4444:8000"
|
||||
... )
|
||||
...
|
||||
>>> url.hostname
|
||||
'fdc8'
|
||||
>>> try:
|
||||
... url.port
|
||||
... except ValueError as error:
|
||||
... print(error)
|
||||
...
|
||||
Port could not be cast to integer value as 'bf8b:e62c:abcd:1111:2222:3333:4444:8000'
|
||||
```
|
||||
|
||||
Since IPv6 addresses use a "colon-hex" format with [hexadecimal] fields
|
||||
separated by `:` we can't tell a port apart from a normal field. Notice in the
|
||||
example above that the hostname is truncated after the first `:`, not the one
|
||||
just before `8000`.
|
||||
|
||||
Fortunately, the spec for URLs recognizes this ambiguity and gives us a way to
|
||||
handle it. [RFC 2732 (_Format for Literal IPv6 Addresses in URL's_)][RFC 2732]
|
||||
says
|
||||
|
||||
> To use a literal IPv6 address in a URL, the literal address should be
|
||||
> enclosed in "[" and "]" characters.
|
||||
|
||||
Update our example above to include `[` and `]` and voilà! It just works.
|
||||
|
||||
```python
|
||||
>>> url = urllib.parse.urlparse(
|
||||
... "https://[fdc8:bf8b:e62c:abcd:1111:2222:3333:4444]:8000"
|
||||
... )
|
||||
...
|
||||
>>> (url.hostname, url.port)
|
||||
('fdc8:bf8b:e62c:abcd:1111:2222:3333:4444', 8000)
|
||||
```
|
||||
|
||||
Armed with that knowledge we can dive into the problem. 🤿
|
||||
|
||||
## Works On My Machine
|
||||
|
||||
A few months ago a co-worker of mine wrote a seemingly innocuous function.
|
||||
|
||||
```python
|
||||
from ipaddress import ip_address
|
||||
|
||||
|
||||
def safe_host(host):
|
||||
"""Surround `host` with brackets if it is an IPv6 address."""
|
||||
try:
|
||||
if ip_address(host).version == 6:
|
||||
return "[{}]".format(host)
|
||||
except ValueError:
|
||||
pass
|
||||
return host
|
||||
```
|
||||
|
||||
Elsewhere in the code it was invoked something like this, so that hostnames,
|
||||
IPv4 addresses, and IPv6 addresses could all be safely interpolated.
|
||||
|
||||
```python
|
||||
url = "https://{host}:8000/some/path/".format(host=safe_host(host))
|
||||
```
|
||||
|
||||
Since my co-worker is awesome they wrote tests to validate their code. ✅
|
||||
|
||||
```python
|
||||
def test_safe_host_with_hostname():
|
||||
"""Hostnames should be unchanged."""
|
||||
assert safe_host("node.example.com") == "node.example.com"
|
||||
|
||||
|
||||
def test_safe_host_with_ipv4_address():
|
||||
"""IPv4 addresses should be unchanged."""
|
||||
assert safe_host("192.168.0.1") == "192.168.0.1"
|
||||
|
||||
|
||||
def test_safe_host_with_ipv6_address():
|
||||
"""IPv6 addresses should be surrounded by brackets."""
|
||||
assert (
|
||||
safe_host("fdc8:bf8b:e62c:abcd:1111:2222:3333:4444")
|
||||
== "[fdc8:bf8b:e62c:abcd:1111:2222:3333:4444]"
|
||||
)
|
||||
```
|
||||
|
||||
Thank goodness they did. The Python 2 tests failed ([don't look at me like
|
||||
that][drop-python2] 😒).
|
||||
|
||||
```ini
|
||||
✖ FAIL py27 in 1.83 seconds
|
||||
✔ OK py36 in 2.82 seconds
|
||||
✔ OK py37 in 2.621 seconds
|
||||
✔ OK py38 in 2.524 seconds
|
||||
✔ OK py39 in 2.461 seconds
|
||||
```
|
||||
|
||||
Both the hostname and IPv6 address tests failed. But _**why**_ did they fail?
|
||||
And why did the Python 3 tests pass? 🤔
|
||||
|
||||
We'll start with the hostname failure and try to isolate the bug.
|
||||
|
||||
```python
|
||||
E AssertionError: assert '[node.example.com]' == 'node.example.com'
|
||||
E - [node.example.com]
|
||||
E ? - -
|
||||
E + node.example.com
|
||||
```
|
||||
|
||||
The failure says `node.example.com` was surrounded by brackets, but that's
|
||||
only supposed to happen for IPv6 addresses! Let's crack open a Python 2
|
||||
interpreter for a quick sanity check.
|
||||
|
||||
```python
|
||||
>>> ipaddress.ip_address("node.example.com").version
|
||||
6
|
||||
```
|
||||
|
||||
<img src='confused-jeff-bridges.webp' alt='Confused Jeff Bridges' style='display: block; width: 100%;'>
|
||||
|
||||
## What On Htrae?
|
||||
|
||||
If, like Jeff Bridges, you were confused by that result, _relax_. We're
|
||||
probably not in a [Bizarro World] where `node.example.com` is a valid IPv6
|
||||
address. There must be an explanation for this behavior.
|
||||
|
||||
Things start to become a little more clear when we see the result of the
|
||||
[`ip_address()`][ip_address] function for ourselves.
|
||||
|
||||
```python
|
||||
>>> ipaddress.ip_address("node.example.com")
|
||||
IPv6Address(u'6e6f:6465:2e65:7861:6d70:6c65:2e63:6f6d')
|
||||
```
|
||||
|
||||
At first glance that looks like madness. Python 3 behaves in an entirely
|
||||
different manner.
|
||||
|
||||
```python
|
||||
>>> try:
|
||||
... ipaddress.ip_address("node.example.com")
|
||||
... except ValueError as error:
|
||||
... print(error)
|
||||
...
|
||||
'node.example.com' does not appear to be an IPv4 or IPv6 address
|
||||
```
|
||||
|
||||
Python 3 knows that's not an IPv6 address, so why doesn't Python 2? The answer
|
||||
is in how differently the two Python versions handle text.
|
||||
|
||||
## Text Is Hard
|
||||
|
||||
Computers don't operate on text as humans think of it. They operate on numbers.
|
||||
That's part of why we have IP addresses to begin with. In order to represent
|
||||
human-readable text with computers we had to assign meaning to the numbers.
|
||||
Thus, [ASCII] was born.
|
||||
|
||||
ASCII is a [character encoding], which means it specifies how to interpret
|
||||
[bytes][byte] as text we understand (provided you speak English). So, when your
|
||||
computer sees `01101110` in [binary] (`110` in [decimal]) you see `n` because
|
||||
that's what ASCII says it is.
|
||||
|
||||
You can see the number to text conversion in action right in the Python
|
||||
interpreter.
|
||||
|
||||
```python
|
||||
>>> ord("n")
|
||||
110
|
||||
>>> chr(110)
|
||||
'n'
|
||||
```
|
||||
|
||||
In fact, it doesn't matter what numbering system you use. If you specify
|
||||
binary, [octal], decimal, hexadecimal, whatever... If it can be understood as
|
||||
the right integer it will be displayed correctly.
|
||||
|
||||
```python
|
||||
>>> chr(0b01101110)
|
||||
'n'
|
||||
>>> chr(0o156)
|
||||
'n'
|
||||
>>> chr(110)
|
||||
'n'
|
||||
>>> chr(0x6e)
|
||||
'n'
|
||||
```
|
||||
|
||||
Neat, but what does that information do for us?
|
||||
|
||||
## It's Numbers All The Way Down
|
||||
|
||||
Just for giggles, humor me and let's look at the character-number translations
|
||||
for `node.example.com`. We'll leave out binary and octal, because they make
|
||||
this table uglier than it already is.
|
||||
|
||||
<table>
|
||||
<tbody>
|
||||
<tr>
|
||||
<th scope="row">Character</th>
|
||||
<td>n</td>
|
||||
<td>o</td>
|
||||
<td>d</td>
|
||||
<td>e</td>
|
||||
<td>.</td>
|
||||
<td>e</td>
|
||||
<td>x</td>
|
||||
<td>a</td>
|
||||
<td>m</td>
|
||||
<td>p</td>
|
||||
<td>l</td>
|
||||
<td>e</td>
|
||||
<td>.</td>
|
||||
<td>c</td>
|
||||
<td>o</td>
|
||||
<td>m</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th scope="row">Decimal</th>
|
||||
<td>110</td>
|
||||
<td>111</td>
|
||||
<td>100</td>
|
||||
<td>101</td>
|
||||
<td>46</td>
|
||||
<td>101</td>
|
||||
<td>120</td>
|
||||
<td>97</td>
|
||||
<td>109</td>
|
||||
<td>112</td>
|
||||
<td>108</td>
|
||||
<td>101</td>
|
||||
<td>46</td>
|
||||
<td>99</td>
|
||||
<td>111</td>
|
||||
<td>109</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th scope="row">Hexadecimal</th>
|
||||
<td>6e</td>
|
||||
<td>6f</td>
|
||||
<td>64</td>
|
||||
<td>65</td>
|
||||
<td>2e</td>
|
||||
<td>65</td>
|
||||
<td>78</td>
|
||||
<td>61</td>
|
||||
<td>6d</td>
|
||||
<td>70</td>
|
||||
<td>6c</td>
|
||||
<td>65</td>
|
||||
<td>2e</td>
|
||||
<td>63</td>
|
||||
<td>6f</td>
|
||||
<td>6d</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
Hey, hold on a second... If you tilt your head sideways and squint that last
|
||||
row looks kinda like an IPv6 address, doesn't it?
|
||||
|
||||
We should verify, just to be absolutely certain. You've still got that Python 2
|
||||
interpreter open, right?
|
||||
|
||||
```python
|
||||
>>> # Convert the characters in the hostname to hexadecimal.
|
||||
>>> hostname = "node.example.com"
|
||||
>>> hostname_as_hexadecimal = "".join(hex(ord(c))[2:] for c in hostname)
|
||||
>>> hostname_as_hexadecimal
|
||||
'6e6f64652e6578616d706c652e636f6d'
|
||||
>>>
|
||||
>>> # Convert the "IP address" to text.
|
||||
>>> address = ipaddress.ip_address(hostname)
|
||||
>>> str(address)
|
||||
'6e6f:6465:2e65:7861:6d70:6c65:2e63:6f6d'
|
||||
>>>
|
||||
>>> # Remove the colons from that text.
|
||||
>>> address_without_colons = str(address).replace(":", "")
|
||||
>>> address_without_colons
|
||||
'6e6f64652e6578616d706c652e636f6d'
|
||||
>>>
|
||||
>>> # Compare the results and see they're equal.
|
||||
>>> hostname_as_hexadecimal == address_without_colons
|
||||
True
|
||||
```
|
||||
|
||||
Sure enough, when you boil them both down to numbers they're the same mess of
|
||||
hexadecimal.
|
||||
|
||||
## The Belly Of The Beast
|
||||
|
||||
If we dig into the source code for the Python 2 version of the
|
||||
[`ipaddress`][ipaddress2] module we ultimately come to a
|
||||
[curious set of lines].
|
||||
|
||||
```python
|
||||
# Constructing from a packed address
|
||||
if isinstance(address, bytes):
|
||||
self._check_packed_address(address, 16)
|
||||
bvs = _compat_bytes_to_byte_vals(address)
|
||||
self._ip = _compat_int_from_byte_vals(bvs, 'big')
|
||||
return
|
||||
```
|
||||
|
||||
It turns out that, under certain conditions, the `ipaddress` module can create
|
||||
IPv6 addresses from raw bytes. My assumption is that it offers this behavior as
|
||||
a convenient way to parse IP addresses from data fresh off the [wire].
|
||||
|
||||
Does `node.example.com` meet those certain conditions? You bet it does. Because
|
||||
we're using Python 2 it's just `bytes` and it happens to be 16 characters long.
|
||||
|
||||
```python
|
||||
>>> isinstance("node.example.com", bytes)
|
||||
True
|
||||
>>> # `self._check_packed_address` basically just checks how long it is.
|
||||
>>> len("node.example.com") == 16
|
||||
True
|
||||
```
|
||||
|
||||
The rest of the `ipaddress` lines say to interpret the sequence of bytes as a
|
||||
[big-endian][endianness] integer. That's [magic][struct unpacking] best left
|
||||
for another blog post, but the gist is that hexadecimal interpretation of
|
||||
`node.example.com` is condensed into a single, **huge** number.
|
||||
|
||||
```python
|
||||
>>> int("6e6f64652e6578616d706c652e636f6d", 16)
|
||||
146793460745001871434687145741037825901L
|
||||
```
|
||||
|
||||
That's an absolutely massive number, but not so massive it won't fit within the
|
||||
[IPv6 address space].
|
||||
|
||||
```python
|
||||
>>> ip_address(146793460745001871434687145741037825901L)
|
||||
IPv6Address(u'6e6f:6465:2e65:7861:6d70:6c65:2e63:6f6d')
|
||||
```
|
||||
|
||||
As it turns out, if you're liberal in your interpretation, `node.example.com`
|
||||
_can_ be an IPv6 address!
|
||||
|
||||
## You Will Be Reading Meanings
|
||||
|
||||
Obviously that's hogwash. Bizarro might be proud, but that's not what we wanted
|
||||
to happen.
|
||||
|
||||
There's a quote about numbers which is apocryphally attributed to [W.E.B. Du
|
||||
Bois], but that actually comes from [Harold Geneen]'s book,
|
||||
[_Managing_][Managing].
|
||||
|
||||
> When you have mastered the numbers, you will in fact no longer be reading
|
||||
> numbers, any more than you read words when reading a book. You will be
|
||||
> reading meanings.
|
||||
|
||||
Having not read the book I'm probably taking the quote way out of context, but
|
||||
I think it fits our situation well.
|
||||
|
||||
As we've seen above, we can freely convert characters to numbers and back
|
||||
again. The root of our problem is that when we use Python 2 it considers text
|
||||
to be bytes. There's not a deeper, inherent meaning. Maybe the bytes are meant
|
||||
to be ASCII, maybe they're meant to be a long number, maybe they're meant to be
|
||||
an IP address. The interpretation of those bytes is up to us.
|
||||
|
||||
Python 2 doesn't differentiate between bytes and text by default. In fact, the
|
||||
`bytes` type is just an [alias][bytes-str-alias] for `str`.
|
||||
|
||||
```python
|
||||
>>> bytes
|
||||
<type 'str'>
|
||||
>>> bytes is str
|
||||
True
|
||||
```
|
||||
|
||||
To make that even more concrete, see how Python 2 considers `n` to be the same
|
||||
as this sequence of raw bytes.
|
||||
|
||||
```python
|
||||
>>> "n" == b"\x6e"
|
||||
True
|
||||
```
|
||||
|
||||
Our Python 2 code doesn't work the way we want it to because raw bytes can have
|
||||
arbitrary meaning and we haven't told it to use our intended meaning.
|
||||
|
||||
So now we know why Python 2 interprets `node.example.com` as an IPv6 address,
|
||||
but why does Python 3 behave differently? More importantly, how can we
|
||||
reconcile the two?
|
||||
|
||||
|
||||
## 256 Characters Ought To Be Enough For Anybody
|
||||
|
||||
ASCII looked like a good idea in the 1960's. With decades of hindsight we
|
||||
know the 256 characters afforded to us by [Extended ASCII] are insufficient to
|
||||
handle all of the world's writing systems. Thus, [Unicode] was born.
|
||||
|
||||
There are scads of blog posts, Wikipedia articles, and technical documents that
|
||||
will do a better job than I can of explaining Unicode in detail. You should
|
||||
read them if you care to, but here's my gist.
|
||||
|
||||
Unicode is a set of character encodings. [UTF-8] is the dominant encoding.
|
||||
UTF-8 overlaps with ASCII, so ASCII characters are still just one byte. To
|
||||
handle the multitude of other characters, however, multiple bytes can express a
|
||||
single character.
|
||||
|
||||
```python
|
||||
>>> "n".encode("utf-8").hex() # 1 character (U+006E), 1 byte.
|
||||
'6e'
|
||||
>>> "🤿".encode("utf-8").hex() # 1 character (U+1F93F), 4 bytes.
|
||||
'f09fa4bf'
|
||||
>>> "悟り".encode("utf-8").hex() # 2 characters (U+609F, U+308A), 6 bytes.
|
||||
'e6829fe3828a'
|
||||
```
|
||||
|
||||
Every programming language I know of that respects the difference between
|
||||
raw bytes and Unicode text maintains a strict separation between the two
|
||||
datatypes.
|
||||
|
||||
In Python 3 this strict separation is enabled by default. Notice that it
|
||||
doesn't consider `n` and this sequence of raw bytes to be the same thing.
|
||||
|
||||
```python
|
||||
>>> "n" == b"\x6e"
|
||||
False
|
||||
```
|
||||
|
||||
Even better, it doesn't consider `str` and `bytes` to be the same type.
|
||||
|
||||
```python
|
||||
>>> bytes is str
|
||||
False
|
||||
>>> bytes
|
||||
<class 'bytes'>
|
||||
```
|
||||
|
||||
If we can get Python 2 to understand Unicode like Python 3 does, then we can
|
||||
probably fix our bug.
|
||||
|
||||
As an aside, if you want to learn more about how to handle Unicode in Python,
|
||||
check out [Ned Batchelder]'s talk on [_Pragmatic Unicode_][unipain].
|
||||
|
||||
## How Did We Fix It?
|
||||
|
||||
Python 2 does actually know about Unicode, but it considers Unicode text to
|
||||
be separate from "normal" text. At some point in Python 2 history the
|
||||
[`unicode`][python2-unicode] type was bolted onto the side of the language and
|
||||
not enabled by default. Hard to get excited about it, but it does the trick.
|
||||
At least they knew it's a pain to type `unicode()` all the time, so there's a
|
||||
handy literal syntax using a `u` prefix.
|
||||
|
||||
```python
|
||||
>>> unicode("node.example.com") == u"node.example.com"
|
||||
True
|
||||
```
|
||||
|
||||
This is _not_ the best fix, but it did in a pinch. We added a line converting
|
||||
the hostname to Unicode right off the bat. We also applied the same
|
||||
transformation to the line with brackets. This way we always process the
|
||||
hostname as Unicode and we always return a Unicode value.
|
||||
|
||||
```diff
|
||||
def safe_host(host):
|
||||
"""Surround `host` with brackets if it is an IPv6 address."""
|
||||
+ host = u"{}".format(host)
|
||||
try:
|
||||
if ip_address(host).version == 6:
|
||||
- return "[{}]".format(host)
|
||||
+ return u"[{}]".format(host)
|
||||
except ValueError:
|
||||
pass
|
||||
```
|
||||
|
||||
Luckily for us the `u` prefix also works in Python 3 whereas `unicode()` does not
|
||||
(because [all text is Unicode by default][python3-str-type], so the type has no
|
||||
business existing). In Python 3 the `u` is treated as a [no-op].
|
||||
|
||||
The Python 2 interpreter graciously understands the `unicode` type is not just
|
||||
raw `bytes`.
|
||||
|
||||
```python
|
||||
>>> isinstance(u"node.example.com", bytes)
|
||||
False
|
||||
```
|
||||
|
||||
When we use the `unicode` type the `ipaddress` module no longer tries to
|
||||
interpret `node.example.com` as `bytes` and convert those bytes to an IP
|
||||
address. We get just what we expect
|
||||
|
||||
```python
|
||||
>>> try:
|
||||
... ipaddress.ip_address(u"node.example.com")
|
||||
... except ValueError as error:
|
||||
... print(error)
|
||||
...
|
||||
u'node.example.com' does not appear to be an IPv4 or IPv6 address
|
||||
```
|
||||
|
||||
and our tests pass!
|
||||
|
||||
```ini
|
||||
✔ OK py27 in 1.728 seconds
|
||||
✔ OK py36 in 2.775 seconds
|
||||
✔ OK py37 in 2.717 seconds
|
||||
✔ OK py38 in 2.674 seconds
|
||||
✔ OK py39 in 2.506 seconds
|
||||
```
|
||||
|
||||
## Reflection
|
||||
|
||||
I mentioned above that our fix wasn't the best. Given more time, how can we do
|
||||
better?
|
||||
|
||||
<p id='drop-python2'>The first (and best) solution here is to
|
||||
<a href='https://python3statement.org/'>drop Python 2 support</a>. It's 2020
|
||||
now and Python 2 is officially no longer supported. The original code worked on
|
||||
Python 3. The best long-term decision is to migrate the code to run on Python 3
|
||||
<b>only</b> and avoid the hassle of Python 2 maintenance. Unfortunately many of
|
||||
the people running this code still depend on it working on Python 2, so we'll
|
||||
have to make that transition gracefully.</p>
|
||||
|
||||
If a migration away from Python 2 isn't possible in the near-term, the next
|
||||
best thing to do is update our code so that it uses a compatibility layer like
|
||||
[`future`][future] or [`six`][six]. Those libraries are designed to modernize
|
||||
Python 2 and help smooth over issues like this one.
|
||||
|
||||
It also wouldn't hurt for us to take a page from [Alexis King]'s
|
||||
[_Parse, don't validate_][parse-dont-validate] school of thought. When the
|
||||
hostname enters our program via user input it should **immediately** be
|
||||
converted to the `unicode` type (or maybe even an IP address type) so we don't
|
||||
end up solving this problem in several different places throughout the code.
|
||||
|
||||
Finally, though our program doesn't currently handle any hostnames in languages
|
||||
other than English, it's probably best to be thinking in Unicode anyway. Again,
|
||||
it's 2020 and [internationalized domain names] like [https://Яндекс.рф][yandex]
|
||||
are a thing.
|
||||
|
||||
If you made it this far, thanks for reading. It was fun to turn a brief
|
||||
debugging session with my co-worker into a treatise on the perils of Python 2
|
||||
and the value of Unicode. See you next year! 😂
|
||||
|
||||
[URL]: https://en.wikipedia.org/wiki/URL
|
||||
[standard ports]: https://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers
|
||||
[port]: https://en.wikipedia.org/wiki/Port_(computer_networking)
|
||||
[hostname]: https://en.wikipedia.org/wiki/Hostname
|
||||
[IPv4]: https://en.wikipedia.org/wiki/IPv4#Addressing
|
||||
[Python]: https://www.python.org/
|
||||
[IPv6]: https://en.wikipedia.org/wiki/IPv6#Addressing
|
||||
[hexadecimal]: https://en.wikipedia.org/wiki/Hexadecimal
|
||||
[RFC 2732]: https://www.ietf.org/rfc/rfc2732.txt
|
||||
[drop-python2]: /#drop-python2
|
||||
[Bizarro World]: https://en.wikipedia.org/wiki/Bizarro_World
|
||||
[ip_address]: https://github.com/python/cpython/blob/v3.9.0/Lib/ipaddress.py#L27-L54
|
||||
[ASCII]: https://en.wikipedia.org/wiki/ASCII
|
||||
[character encoding]: https://en.wikipedia.org/wiki/Character_encoding
|
||||
[byte]: https://en.wikipedia.org/wiki/Byte
|
||||
[binary]: https://en.wikipedia.org/wiki/Binary_number
|
||||
[decimal]: https://en.wikipedia.org/wiki/Decimal
|
||||
[octal]: https://en.wikipedia.org/wiki/Octal
|
||||
[W.E.B. Du Bois]: https://en.wikipedia.org/wiki/W._E._B._Du_Bois
|
||||
[Harold Geneen]: https://en.wikipedia.org/wiki/Harold_Geneen
|
||||
[Managing]: https://en.wikiquote.org/wiki/Harold_Geneen
|
||||
[bytes-str-alias]: https://docs.python.org/3/whatsnew/2.6.html#pep-3112-byte-literals
|
||||
[Extended ASCII]: https://en.wikipedia.org/wiki/ASCII#8-bit_codes
|
||||
[Unicode]: https://en.wikipedia.org/wiki/Unicode
|
||||
[UTF-8]: https://en.wikipedia.org/wiki/UTF-8
|
||||
[Ned Batchelder]: https://nedbatchelder.com/site/aboutned.html
|
||||
[unipain]: https://nedbatchelder.com/text/unipain.html
|
||||
[no-op]: https://en.wikipedia.org/wiki/NOP_(code)
|
||||
[ipaddress2]: https://github.com/phihag/ipaddress/blob/v1.0.23/ipaddress.py
|
||||
[curious set of lines]: https://github.com/phihag/ipaddress/blob/v1.0.23/ipaddress.py#L2026-L2031
|
||||
[wire]: https://en.wikipedia.org/wiki/Wire_data
|
||||
[endianness]: https://en.wikipedia.org/wiki/Endianness
|
||||
[struct unpacking]: https://docs.python.org/3.9/library/struct.html#struct.unpack
|
||||
[IPv6 address space]: https://en.wikipedia.org/wiki/IPv6#Larger_address_space
|
||||
[python2-unicode]: https://docs.python.org/2.7/library/functions.html#unicode
|
||||
[python3-str-type]: https://docs.python.org/3/library/stdtypes.html#text-sequence-type-str
|
||||
[future]: https://python-future.org/
|
||||
[six]: https://six.readthedocs.io/
|
||||
[Alexis King]: https://lexi-lambda.github.io/about.html
|
||||
[parse-dont-validate]: https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/
|
||||
[internationalized domain names]: https://en.wikipedia.org/wiki/Internationalized_domain_name
|
||||
[yandex]: https://Яндекс.рф
|
||||
@@ -0,0 +1,120 @@
|
||||
+++
|
||||
title = "Setting up Obsidian sync in Windows"
|
||||
description = "Syncing Obsidian in Windows with Windows Task Scheduler with git"
|
||||
author = "Ian Andwati"
|
||||
date = 2023-10-01
|
||||
[taxonomies]
|
||||
tags = ["obsidian", "git","sync", "windows", "cron", "task scheduler", "powershell"]
|
||||
+++
|
||||
|
||||
Syncing Obsidian in Windows with Windows Task Scheduler and git
|
||||
|
||||
<!-- more -->
|
||||
|
||||
[Obsidian](https://obsidian.md/) is a powerful note-taking and knowledge management tool that has gained a dedicated following among researchers, writers, and knowledge workers. To make the most of Obsidian, it's essential to keep your notes synchronized across devices and ensure your data is regularly backed up.Obsidian is the private and flexible writing app that adapts to the way you think.
|
||||
|
||||
Obsidian is free for personal use.It does not charge based on features or usage.You only pay if you use Obsidian commercially.
|
||||
|
||||
In this article, we'll guide you through the process of setting up automated synchronization for your Obsidian vault on Windows using Windows Task Scheduler and a PowerShell script.[^note]
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Before we dive into setting up the synchronization task, ensure you have the following prerequisites in place:
|
||||
|
||||
1. **Obsidian Installed:** Make sure Obsidian is [installed](https://obsidian.md/download) on your Windows computer, and you have an existing Obsidian vault that you want to sync.
|
||||
|
||||
1. **Git Installed:** You'll need Git installed and configured with your Obsidian vault's repository. If you haven't already done this, you can download Git from the [official website](https://git-scm.com/downloads) and follow the installation instructions.
|
||||
|
||||
1. **PowerShell Script:** The provided PowerShell script, sync.ps1, should be located in your Obsidian vault folder.
|
||||
|
||||
## Understanding the PowerShell Script
|
||||
|
||||
The provided PowerShell script, sync.ps1, is the heart of our synchronization task. It automates the process of committing changes in your Obsidian vault's Git repository and pushing them to your remote repository (e.g., GitHub or GitLab). Here's a brief overview of the script:
|
||||
|
||||
```ps1
|
||||
# Define the folder path
|
||||
$FolderPath = "C:\path\to\obsidian-vault"
|
||||
|
||||
# Change directory to the repository folder
|
||||
Set-Location -Path $FolderPath
|
||||
|
||||
# Check if the working tree is clean
|
||||
$IsClean = (git diff-index --quiet HEAD --) -or (git diff-files --quiet)
|
||||
|
||||
if ($IsClean) {
|
||||
Write-Host "Working tree is clean. Skipping commit and pushing changes."
|
||||
git push
|
||||
} else {
|
||||
# Get the current date in the desired format (Day - Date)
|
||||
$CommitMessage = "$(Get-Date -Format 'dddd - yyyy-MM-dd HH:mm:ss')"
|
||||
|
||||
# Commit the changes
|
||||
git add .
|
||||
git commit -m $CommitMessage
|
||||
|
||||
# Push the changes
|
||||
git push
|
||||
}
|
||||
```
|
||||
|
||||
This script does the following:
|
||||
|
||||
- Sets the folder path to your Obsidian vault.
|
||||
- Checks if there are any changes to commit. If the vault is clean (no changes), it skips the commit and only pushes changes.
|
||||
- If there are changes, it commits them with a timestamped message and then pushes them to the remote repository.
|
||||
|
||||
## Setting up Windows Task Scheduler
|
||||
|
||||
Now that we have our PowerShell script ready, let's set up a scheduled task using Windows Task Scheduler to automate the synchronization process. Follow these steps:
|
||||
|
||||
1. **Open an Elevated PowerShell Prompt:** To run the necessary commands, open PowerShell as an administrator.
|
||||
|
||||
1. **Define Task Variables:** Execute the following commands to define variables for your task:
|
||||
|
||||
```ps1
|
||||
$ScriptPath = "C:\path\to\obsidian-vault\sync.ps1"
|
||||
$TaskName = "obsidian-sync"
|
||||
$UserAccount = "$env:USERDOMAIN\$env:USERNAME"
|
||||
```
|
||||
|
||||
Replace $ScriptPath with the actual path to your sync.ps1 script, $TaskName with a name for your task.
|
||||
|
||||
1. **Configure Task Trigger:** Set up a trigger for your task. In this example, we'll schedule the task to run daily at 9:00 PM:
|
||||
|
||||
```ps1
|
||||
$Trigger = New-ScheduledTaskTrigger -Daily -At '9:00 PM'
|
||||
```
|
||||
|
||||
You can adjust the schedule according to your preference.
|
||||
|
||||
1. Define Task Principal and Settings: Configure the task's principal (the user account under which the task will run) and settings:
|
||||
|
||||
```ps1
|
||||
$Principal = New-ScheduledTaskPrincipal -UserId $UserAccount -RunLevel Highest
|
||||
$Settings = New-ScheduledTaskSettingsSet -AllowStartIfOnBatteries -RunOnlyIfNetworkAvailable -WakeToRun
|
||||
```
|
||||
|
||||
These settings ensure that the task runs with the necessary privileges and conditions.
|
||||
|
||||
1. **Define Task Action:** Specify the action that the task will perform, which is running the PowerShell script:
|
||||
|
||||
```ps1
|
||||
$Action = New-ScheduledTaskAction -Execute 'PowerShell.exe' -Argument "-ExecutionPolicy Bypass -WindowStyle Hidden -File `"$ScriptPath`""
|
||||
```
|
||||
|
||||
This action runs PowerShell with the provided script.
|
||||
|
||||
1. **Create the Scheduled Task:** Finally, create the scheduled task and register it with Windows Task Scheduler:
|
||||
|
||||
```ps1
|
||||
$Task = New-ScheduledTask -Action $Action -Trigger $Trigger -Principal $Principal -Settings $Settings
|
||||
Register-ScheduledTask -TaskName $TaskName -InputObject $Task
|
||||
```
|
||||
|
||||
The Obsidian synchronization task is now set up and ready to run automatically based on the schedule you defined.
|
||||
|
||||
# Inspiration
|
||||
|
||||
When looking up on how to implement this I found some breakthrough from this [stack overflow](https://stackoverflow.com/questions/7195503/setting-up-a-cron-job-in-windows) post. There are suggestions on how to do this using cron jobs in wsl. They are worth a look at. This article on [powershell](https://learn.microsoft.com/en-us/powershell/module/scheduledtasks/new-scheduledtask?view=windowsserver2022-ps) with windows server was also helpful.
|
||||
|
||||
[^note]: This method is only optimal for desktop sync. To sync across devices obsidian has a paid plan that has some extra features Such as Sync notes across devices, End-to-end encryption, Version history, Priority email support. Consider checking it out at [https://obsidian.md/pricing](https://obsidian.md/pricing)
|
||||
|
Before Width: | Height: | Size: 20 KiB |
|
Before Width: | Height: | Size: 20 KiB |
|
Before Width: | Height: | Size: 143 KiB |
|
Before Width: | Height: | Size: 141 KiB |
@@ -1,316 +0,0 @@
|
||||
+++
|
||||
title = "Resolving A DNS Issue"
|
||||
description = "Resolving my DNS issue."
|
||||
date = 2018-06-24T13:21:00-07:00
|
||||
[taxonomies]
|
||||
tags = ["DNS", "HTTP", "Slow", "Internet"]
|
||||
+++
|
||||
Haha. Get it? _Resolving_ a DNS issue. OK, that was bad. You don't have to read
|
||||
anymore, but I'm `SOA` into this. You might even say I'm in the zone. I think
|
||||
it's gonna be `A` great read, so consider sticking around, 'cuz there's no
|
||||
TLD;R.
|
||||
|
||||
<!-- more -->
|
||||
|
||||
## Symptoms
|
||||
|
||||
A few months ago I switched ISPs to try to support a smaller regional ISP
|
||||
instead of one of the [monopolistic][comcast] [behemoths][warner]. The
|
||||
transition was largely pain free, but after a little while I noticed that
|
||||
something wasn't right on the new network. Periodically my web browsing would
|
||||
be much slower, but it didn't seem to be specific to any particular site.
|
||||
Furthermore, reloading a site I was having a problem with was usually without
|
||||
issue.
|
||||
|
||||
I let this go on for longer than I'm comfortable admitting, but finally got
|
||||
fed up when I was trying to `curl` pages from the same site in a loop. I
|
||||
noticed that the same pages took different amounts of time to fetch between
|
||||
runs of the loop, which seemed very wrong to me since they were static pages.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
My first step in narrowing down the problem was to pick a website known for its
|
||||
speed and reliability see if I had problems there. I `curl`ed
|
||||
[google.com][google] in a loop every 10 seconds and sure enough, every
|
||||
so often the problem would happen there too. The connection would just hang
|
||||
for a few seconds, but eventually complete.
|
||||
|
||||
Next, I wondered how I could narrow it down further to make sure the issue wasn't
|
||||
caused on my home network. Then I remembered that my gateway could be
|
||||
accessed by its IP of [192.168.1.1][router-ip] and the hostname
|
||||
[router.asus.com][asus]. I pointed `curl` at my router's hostname. The same
|
||||
problem was happening with a domain that resolved to my local network!
|
||||
|
||||
OK, so it didn't matter whether I was leaving my network or not. Traffic was
|
||||
_slooow_ either way. I changed my `curl` loop to point at my router by its IP.
|
||||
No issues whatsoever. At this point I recalled the age-old sysadmin wisdom.
|
||||
It's **always** DNS!
|
||||
|
||||
<figure>
|
||||
<img src='its-always-dns.svg' alt="It's Always DNS">
|
||||
<figcaption>
|
||||
It's Always DNS. Converted to SVG from a <a href='https://www.reddit.com/r/sysadmin/comments/34ag51/its_always_dns/'>Reddit post</a> by <a href="https://www.reddit.com/u/tehrabbitt">u/tehrabbitt</a>. <a href='its-always-dns.svg'>View full size.</a>
|
||||
</figcaption>
|
||||
</figure>
|
||||
|
||||
## Gathering Data
|
||||
|
||||
Clearly this was an intermittent DNS issue. I could have gone right into fixing
|
||||
the problem, but I wanted to know just how bad it was. In order to do that I
|
||||
needed data. I collected timing information for simultaneous HTTP GET requests
|
||||
resolving by IP address and by hostname for the same domain over a 2 hour
|
||||
period with the help of this script.
|
||||
|
||||
```zsh
|
||||
#!/usr/bin/env zsh
|
||||
|
||||
# 720 * 10s = 2hrs
|
||||
: ${INTERVAL=10}
|
||||
: ${NUM_TESTS=720}
|
||||
|
||||
: ${ROUTER_IP="192.168.1.1"}
|
||||
: ${ROUTER_HOSTNAME="router.asus.com"}
|
||||
|
||||
: ${OUTPUT_FILE="./http-test-results.csv"}
|
||||
|
||||
function now() {
|
||||
# ISO 8601 FTW!
|
||||
date -u "+%Y-%m-%dT%H:%M:%SZ"
|
||||
}
|
||||
|
||||
function http-test() {
|
||||
local host=$1
|
||||
/usr/bin/time -f "%e" curl -s $host > /dev/null
|
||||
}
|
||||
|
||||
function run-tests() {
|
||||
# Create CSV header.
|
||||
echo "Datetime,Seconds for HTTP GET by IP,Seconds for HTTP GET by Hostname"
|
||||
|
||||
# A sample CSV row will look like
|
||||
# 2018-03-26T03:37:29Z,0.01,0.02
|
||||
|
||||
# For each iteration spit out the current time, how long it takes to curl
|
||||
# with an IP address, and how long it takes to curl with a hostname.
|
||||
for t in {1..${NUM_TESTS}}
|
||||
do
|
||||
# Redirect stderr to stdout with 2>&1 to capture GNU time's output.
|
||||
echo "$(now),$(http-test ${ROUTER_IP} 2>&1),$(http-test ${ROUTER_HOSTNAME} 2>&1)"
|
||||
sleep ${INTERVAL}
|
||||
done
|
||||
}
|
||||
|
||||
run-tests >> ${OUTPUT_FILE}
|
||||
```
|
||||
|
||||
That was going to give me the data I needed, but reading raw CSVs is a pain.
|
||||
I don't do graphing very often, but I figured visualization would help, so I
|
||||
did a quick survey of Python graphing libraries and found the [Pygal][pygal]
|
||||
library. It turned out to be much more flexible than Matplotlib for my needs
|
||||
and gave me great looking SVGs customized to match the style of this site.
|
||||
|
||||
```python
|
||||
#!/usr/bin/env python3.6
|
||||
|
||||
import sys
|
||||
from typing import Iterator
|
||||
|
||||
import pandas as pd
|
||||
from pygal import Line
|
||||
from pygal.style import Style
|
||||
|
||||
|
||||
class GruvboxStyle(Style):
|
||||
""" A gruvbox-inspired Pygal style. """
|
||||
|
||||
background = '#282828'
|
||||
plot_background = '#1d2021'
|
||||
foreground = '#fdf4c1'
|
||||
foreground_strong = '#fdf4c1'
|
||||
foreground_subtle = '#fdf4c1'
|
||||
colors = ('#8ec07c', '#fa5c4b')
|
||||
|
||||
|
||||
def dilute_datetimes(datetimes: pd.Series, factor: int) -> Iterator[str]:
|
||||
""" Lots of datetimes overlap and become unreadable, make some space. """
|
||||
dilute = lambda t: t[1] if t[0] % factor == 0 else ''
|
||||
yield from map(dilute, enumerate(datetimes))
|
||||
|
||||
|
||||
def generate_chart(data: pd.DataFrame) -> Line:
|
||||
line_chart = Line(
|
||||
js=(), # The tooltips are really nice, but I don't want any JS.
|
||||
style=GruvboxStyle,
|
||||
x_label_rotation=30
|
||||
)
|
||||
|
||||
# Water those datetimes down so they don't overlap and we can read them!
|
||||
datetimes = data['Datetime']
|
||||
dilution_factor = datetimes.shape[0] // 10
|
||||
datetimes = dilute_datetimes(datetimes, factor=dilution_factor)
|
||||
|
||||
line_chart.title = 'HTTP GET by IP vs. HTTP GET by Hostname'
|
||||
line_chart.y_title = 'Seconds'
|
||||
line_chart.x_labels = datetimes
|
||||
line_chart.add(
|
||||
title='By IP',
|
||||
values=data['Seconds for HTTP GET by IP']
|
||||
)
|
||||
line_chart.add(
|
||||
title='By Hostname',
|
||||
values=data['Seconds for HTTP GET by Hostname']
|
||||
)
|
||||
return line_chart
|
||||
|
||||
|
||||
def main(argv: list) -> None:
|
||||
data = pd.read_csv(sys.argv[1])
|
||||
output = sys.argv[2]
|
||||
chart = generate_chart(data)
|
||||
chart.render_to_file(output)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main(sys.argv)
|
||||
```
|
||||
|
||||
Now I could run a few simple commands and get a chart to help me understand
|
||||
what I was dealing with.
|
||||
|
||||
```zsh
|
||||
#!/usr/bin/env zsh
|
||||
export OUTPUT_FILE=before.csv
|
||||
zsh http-test.zsh && python3.6 chart.py $OUTPUT_FILE ${OUTPUT_FILE:r}.svg
|
||||
```
|
||||
|
||||
I ended up truncating the results for the graphs below to only show 30 minutes
|
||||
of the 2 hour data to help readability, but the data looks just about the same
|
||||
anyway.
|
||||
|
||||
<figure>
|
||||
<img src='before.svg' alt="HTTP GET by IP vs. HTTP GET by Hostname Before the Fix">
|
||||
<figcaption>
|
||||
HTTP GET by IP vs. HTTP GET by Hostname before the fix. <a href='before.svg'>View full size.</a>
|
||||
</figcaption>
|
||||
</figure>
|
||||
|
||||
**Wow!** That's _really_ bad. You can't even see the _By IP_ series because
|
||||
it's dwarfed by these enormous outliers from _By Hostname_. These were local
|
||||
DNS requests. There's no reason they should have been taking **greater than 5
|
||||
seconds** to
|
||||
complete.
|
||||
|
||||
## Resolution
|
||||
|
||||
I logged into my router's administrative console and sure enough, right there
|
||||
in the logs was a DNS error.
|
||||
|
||||
```
|
||||
Mar 25 21:44:03 dnsmasq[31700]: nameserver 208.76.152.1 refused to do a recursive query
|
||||
```
|
||||
|
||||
I was still using the default upstream DNS given by my ISP and it was refusing
|
||||
[recursive DNS queries][recursive-dns]. My OS wasn't running a caching DNS
|
||||
resolver locally, so I was totally at the mercy of my router. When the router
|
||||
went out to lunch trying to satisfy a DNS request my connections did too.
|
||||
|
||||
Curiously, only one of the two nameservers my router was configured to use
|
||||
appeared in the logs with that error message. Not knowing exactly how the
|
||||
router manufacturer configured DNSMasq or how the upstream DNS servers were
|
||||
configured my working hypothesis is that the one refusing recursive queries
|
||||
was the primary and requests that weren't cached always had to go to two
|
||||
servers.
|
||||
|
||||
My assumption is that DNSMasq is pretty badly misconfigured on my router as I
|
||||
wouldn't expect [router.asus.com][asus] to even need a recursive query.
|
||||
Ultimately I should probably run my own local DNS. Maybe I'll even take
|
||||
[trust-dns] for a spin. That would be a long-term project though, so I opted
|
||||
instead to simply change my nameservers to [Google's public ones][google-dns].
|
||||
The problem finally went away once recursive queries were being supported!
|
||||
|
||||
## Measuring Success
|
||||
|
||||
Once I was confident I'd fixed the problem I ran the same chart generation
|
||||
commands as before, being careful to set `OUTPUT_FILE="after.csv"` so as not to
|
||||
overwrite my previous data.
|
||||
|
||||
<figure>
|
||||
<img src='after.svg' alt="HTTP GET by IP vs. HTTP GET by Hostname After the Fix">
|
||||
<figcaption>
|
||||
HTTP GET by IP vs. HTTP GET by Hostname after the fix. <a href='after.svg'>View full size.</a>
|
||||
</figcaption>
|
||||
</figure>
|
||||
|
||||
Much better! Even the most extreme request was under two tenths of a second.
|
||||
Just how much did the situation improve though? I wrote one last script to find
|
||||
out.
|
||||
|
||||
```zsh
|
||||
#!/usr/bin/env zsh
|
||||
function percent_failures() {
|
||||
local file="${1}"
|
||||
local pattern="${2}"
|
||||
local num_failures=$(grep $pattern $file | wc -l)
|
||||
local num_results=$(wc -l $file | cut -d\ -f1)
|
||||
echo "scale=4; ${num_failures} / ${num_results} * 100" | bc -l
|
||||
}
|
||||
|
||||
percent_failures before.csv '5\.'
|
||||
percent_failures after.csv '5\.'
|
||||
```
|
||||
|
||||
Prior to fixing the issue 11.66% of all requests took longer than 5 seconds.
|
||||
After applying the fix that dropped to 0.13%. I'd say that's a noticeable
|
||||
improvement!
|
||||
|
||||
#### Bonus
|
||||
|
||||
In case you were curious about how the [_It's Always DNS_][its-always-dns]
|
||||
diagram above was generated, here's the [Graphviz][graphviz] source and `dot`
|
||||
command to reproduce it.
|
||||
|
||||
```dot
|
||||
// File: its-always-dns.dot
|
||||
|
||||
digraph G {
|
||||
graph [ splines=ortho nodesep=2 fontcolor="#fdf4c1" ];
|
||||
node [ style=filled color="#282828" fillcolor="#282828", fontcolor="#fdf4c1" ];
|
||||
edge [ color="#282828" fontcolor="#282828" ];
|
||||
|
||||
is_it_dns [shape=diamond label="Is it DNS?"];
|
||||
check_bad_records [shape=box label="Check DNS for Bad Records"];
|
||||
is_it_resolved [shape=diamond label="Is the Issue Resolved?"];
|
||||
you_missed_something [shape=box label="You Obviously Missed Something"];
|
||||
its_always_dns [shape=box label="It's Always DNS"];
|
||||
|
||||
is_it_dns -> check_bad_records [xlabel="Yes"];
|
||||
check_bad_records -> is_it_resolved;
|
||||
is_it_resolved -> its_always_dns [xlabel="Yes"];
|
||||
is_it_resolved -> you_missed_something [xlabel="No"];
|
||||
you_missed_something -> check_bad_records;
|
||||
you_missed_something -> is_it_dns [xlabel="No"];
|
||||
}
|
||||
```
|
||||
|
||||
```zsh
|
||||
#!/usr/bin/env zsh
|
||||
dot -Tsvg its-always-dns.dot -o its-always-dns.svg
|
||||
```
|
||||
|
||||
In the interest of making this post maximally reproducible I've put all the
|
||||
raw data and scripts used in this post in a
|
||||
[GitHub repo][resolving-a-dns-issue] for anyone who wants to play with them
|
||||
on their own network.
|
||||
|
||||
[comcast]: https://en.wikipedia.org/wiki/Comcast
|
||||
[warner]: https://en.wikipedia.org/wiki/WarnerMedia
|
||||
[google]: https://google.com/
|
||||
[router-ip]: https://192.168.1.1
|
||||
[asus]: https://router.asus.com
|
||||
[graphviz]: https://www.graphviz.org/
|
||||
[pygal]: https://github.com/Kozea/pygal
|
||||
[recursive-dns]: https://unix.stackexchange.com/a/24391/82338
|
||||
[trust-dns]: https://github.com/bluejekyll/trust-dns
|
||||
[google-dns]: https://developers.google.com/speed/public-dns/
|
||||
[its-always-dns]: its-always-dns.svg
|
||||
[resolving-a-dns-issue]: https://github.com/reillysiemens/resolving-a-dns-issue
|
||||
@@ -1,83 +0,0 @@
|
||||
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
|
||||
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
|
||||
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
|
||||
<!-- Generated by graphviz version 2.40.1 (20161225.0304)
|
||||
-->
|
||||
<!-- Title: G Pages: 1 -->
|
||||
<svg width="760pt" height="260pt"
|
||||
viewBox="0.00 0.00 760.33 260.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
|
||||
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 256)">
|
||||
<title>G</title>
|
||||
<polygon fill="#ffffff" stroke="transparent" points="-4,4 -4,-256 756.3341,-256 756.3341,4 -4,4"/>
|
||||
<!-- is_it_dns -->
|
||||
<g id="node1" class="node">
|
||||
<title>is_it_dns</title>
|
||||
<polygon fill="#282828" stroke="#282828" points="668.6908,-252 584.904,-234 668.6908,-216 752.4777,-234 668.6908,-252"/>
|
||||
<text text-anchor="middle" x="668.6908" y="-230.3" font-family="Times,serif" font-size="14.00" fill="#fdf4c1">Is it DNS?</text>
|
||||
</g>
|
||||
<!-- check_bad_records -->
|
||||
<g id="node2" class="node">
|
||||
<title>check_bad_records</title>
|
||||
<polygon fill="#282828" stroke="#282828" points="615.1908,-180 394.1908,-180 394.1908,-144 615.1908,-144 615.1908,-180"/>
|
||||
<text text-anchor="middle" x="504.6908" y="-158.3" font-family="Times,serif" font-size="14.00" fill="#fdf4c1">Check DNS for Bad Records</text>
|
||||
</g>
|
||||
<!-- is_it_dns->check_bad_records -->
|
||||
<g id="edge1" class="edge">
|
||||
<title>is_it_dns->check_bad_records</title>
|
||||
<path fill="none" stroke="#282828" d="M600.1192,-230.5273C600.1192,-230.5273 600.1192,-190.2123 600.1192,-190.2123"/>
|
||||
<polygon fill="#282828" stroke="#282828" points="603.6193,-190.2122 600.1192,-180.2123 596.6193,-190.2123 603.6193,-190.2122"/>
|
||||
<text text-anchor="middle" x="587.6192" y="-199.1698" font-family="Times,serif" font-size="14.00" fill="#282828">Yes</text>
|
||||
</g>
|
||||
<!-- is_it_resolved -->
|
||||
<g id="node3" class="node">
|
||||
<title>is_it_resolved</title>
|
||||
<polygon fill="#282828" stroke="#282828" points="162.6908,-108 -.1911,-90 162.6908,-72 325.5727,-90 162.6908,-108"/>
|
||||
<text text-anchor="middle" x="162.6908" y="-86.3" font-family="Times,serif" font-size="14.00" fill="#fdf4c1">Is the Issue Resolved?</text>
|
||||
</g>
|
||||
<!-- check_bad_records->is_it_resolved -->
|
||||
<g id="edge2" class="edge">
|
||||
<title>check_bad_records->is_it_resolved</title>
|
||||
<path fill="none" stroke="#282828" d="M404.6908,-143.8314C404.6908,-122.502 404.6908,-90 404.6908,-90 404.6908,-90 335.451,-90 335.451,-90"/>
|
||||
<polygon fill="#282828" stroke="#282828" points="335.451,-86.5001 325.451,-90 335.4509,-93.5001 335.451,-86.5001"/>
|
||||
</g>
|
||||
<!-- you_missed_something -->
|
||||
<g id="node4" class="node">
|
||||
<title>you_missed_something</title>
|
||||
<polygon fill="#282828" stroke="#282828" points="670.1908,-36 415.1908,-36 415.1908,0 670.1908,0 670.1908,-36"/>
|
||||
<text text-anchor="middle" x="542.6908" y="-14.3" font-family="Times,serif" font-size="14.00" fill="#fdf4c1">You Obviously Missed Something</text>
|
||||
</g>
|
||||
<!-- is_it_resolved->you_missed_something -->
|
||||
<g id="edge4" class="edge">
|
||||
<title>is_it_resolved->you_missed_something</title>
|
||||
<path fill="none" stroke="#282828" d="M277.5363,-84.3503C277.5363,-67.377 277.5363,-18 277.5363,-18 277.5363,-18 405.1618,-18 405.1618,-18"/>
|
||||
<polygon fill="#282828" stroke="#282828" points="405.1618,-21.5001 415.1618,-18 405.1618,-14.5001 405.1618,-21.5001"/>
|
||||
<text text-anchor="middle" x="297.1739" y="-21.8" font-family="Times,serif" font-size="14.00" fill="#282828">No</text>
|
||||
</g>
|
||||
<!-- its_always_dns -->
|
||||
<g id="node5" class="node">
|
||||
<title>its_always_dns</title>
|
||||
<polygon fill="#282828" stroke="#282828" points="229.6908,-36 95.6908,-36 95.6908,0 229.6908,0 229.6908,-36"/>
|
||||
<text text-anchor="middle" x="162.6908" y="-14.3" font-family="Times,serif" font-size="14.00" fill="#fdf4c1">It's Always DNS</text>
|
||||
</g>
|
||||
<!-- is_it_resolved->its_always_dns -->
|
||||
<g id="edge3" class="edge">
|
||||
<title>is_it_resolved->its_always_dns</title>
|
||||
<path fill="none" stroke="#282828" d="M162.6908,-71.8314C162.6908,-71.8314 162.6908,-46.4133 162.6908,-46.4133"/>
|
||||
<polygon fill="#282828" stroke="#282828" points="166.1909,-46.4132 162.6908,-36.4133 159.1909,-46.4133 166.1909,-46.4132"/>
|
||||
<text text-anchor="middle" x="150.1908" y="-47.9223" font-family="Times,serif" font-size="14.00" fill="#282828">Yes</text>
|
||||
</g>
|
||||
<!-- you_missed_something->is_it_dns -->
|
||||
<g id="edge6" class="edge">
|
||||
<title>you_missed_something->is_it_dns</title>
|
||||
<path fill="none" stroke="#282828" d="M642.6908,-36.0445C642.6908,-36.0445 642.6908,-211.5083 642.6908,-211.5083"/>
|
||||
<polygon fill="#282828" stroke="#282828" points="639.1909,-211.5082 642.6908,-221.5083 646.1909,-211.5083 639.1909,-211.5082"/>
|
||||
<text text-anchor="middle" x="631.6908" y="-127.5764" font-family="Times,serif" font-size="14.00" fill="#282828">No</text>
|
||||
</g>
|
||||
<!-- you_missed_something->check_bad_records -->
|
||||
<g id="edge5" class="edge">
|
||||
<title>you_missed_something->check_bad_records</title>
|
||||
<path fill="none" stroke="#282828" d="M515.1908,-36.2377C515.1908,-36.2377 515.1908,-133.9104 515.1908,-133.9104"/>
|
||||
<polygon fill="#282828" stroke="#282828" points="511.6909,-133.9103 515.1908,-143.9104 518.6909,-133.9104 511.6909,-133.9103"/>
|
||||
</g>
|
||||
</g>
|
||||
</svg>
|
||||
|
Before Width: | Height: | Size: 5.0 KiB |
@@ -0,0 +1,89 @@
|
||||
+++
|
||||
title= "Semantic Versioning(SemVer) and Conventional Commits"
|
||||
description = "Adopting best practices in your git and build workflows."
|
||||
date= 2023-01-24T15:34:14.160Z
|
||||
author = "Ian Andwati"
|
||||
[taxonomies]
|
||||
tags= ["git", "semver","workflow", "conventional commits"]
|
||||
+++
|
||||
|
||||
Adopting best practices in your git and build workflows.
|
||||
|
||||
<!-- more -->
|
||||
|
||||
# What is Semver?
|
||||
|
||||
Semantic versioning or just SemVer is a dependency naming guidline that aims to bring some sanity into naming libraries and packages in software development. It is a standard versioning system created by **Tom Preston Werner** (co-founder of Github) that’s used to communicate what changes are in a release.
|
||||
|
||||
An efficient and smooth versioning becomes crucial for efficient software delivery expecially for projects that expose a public API. This saves the developers the chaos that is **dependency hell**.Dependency hell is a colloquial term for the frustration of some software users who have installed software packages which have dependencies on specific versions of other software packages.
|
||||
|
||||
The dependency issue arises when several packages have dependencies on the same shared packages or libraries, but they depend on different and incompatible versions of the shared packages. If the shared package or library can only be installed in a single version, the user may need to address the problem by obtaining newer or older versions of the dependent packages. This, in turn, may break other dependencies and push the problem to another set of packages.
|
||||
|
||||
## Semantic Versioning Specifications
|
||||
|
||||
### Summary
|
||||
|
||||
Semantic versioning is based on three numbers that indicate the versions of the software and the compatibilities.They are `X.Y.Z` corresponding to `Major.Minor.Patch`
|
||||
|
||||
- **Major Version (X):**Introduces major changes to the code and breaking changes toward backward compatibility
|
||||
- **Minor Version (Y):** These code changes don’t introduce breaking changes. This could be adding a new feature that doesn't break changes but are not exactly bug fixes.
|
||||
- **Patches (Z):** Mainly correspond to bug fixes
|
||||
|
||||
### Examples in common packages
|
||||
|
||||
#### React(v18.2.0)
|
||||
|
||||

|
||||
|
||||
- **Major -> 18**
|
||||
- **Minor -> 2**
|
||||
- **Patch -> 0**
|
||||
|
||||
---
|
||||
|
||||
#### Pipx(v1.1.0)
|
||||
|
||||

|
||||
|
||||
- **Major -> 1**
|
||||
- **Minor -> 1**
|
||||
- **Patch -> 0**
|
||||
|
||||
---
|
||||
|
||||
## Start at v0.1.0 or v1.0.0?
|
||||
|
||||
Convention recommends that in the initial development phase the starting version should be 0.1.0 which makes sense since it’s not in production yet.
|
||||
|
||||
The version should only be bumped to 1.0.0 only if it is being used in production and stable.
|
||||
|
||||
If your software is being used in production, it should probably already be 1.0.0. If you have a stable API on which users have come to depend, you should be 1.0.0. If you’re worrying a lot about backwards compatibility, you should probably already be 1.0.0.
|
||||
|
||||
# Conventional Commits
|
||||
|
||||
Conventional Commits is a convention on top of commit messages that are used to automate version changes. It provides an easy set of rules for creating commit messages and works well with SemVer by describing features, fixes and breaking changes that correspond to SemVer standards.
|
||||
|
||||
## Summary
|
||||
|
||||
The commit message should be structured as follows:
|
||||
|
||||
```
|
||||
<type>[optional scope]: <description>
|
||||
|
||||
[optional body]
|
||||
|
||||
[optional footer(s)]
|
||||
```
|
||||
|
||||
The commit contains the following structural elements, to communicate intent to the consumers of your library:
|
||||
|
||||
1. fix: a commit of the type fix patches a bug in your codebase (this correlates with PATCH in Semantic Versioning).
|
||||
2. feat: a commit of the type feat introduces a new feature to the codebase (this correlates with MINOR in Semantic Versioning).
|
||||
3. BREAKING CHANGE: a commit that has a footer BREAKING CHANGE:, or appends a ! after the type/scope, introduces a breaking API change (correlating with MAJOR in Semantic Versioning). A BREAKING CHANGE can be part of commits of any type.
|
||||
4. types other than fix: and feat: are allowed, for example @commitlint/config-conventional (based on the Angular convention) recommends build:, chore:, ci:, docs:, style:, refactor:, perf:, test:, and others.
|
||||
5. footers other than BREAKING CHANGE: <description> may be provided and follow a convention similar to git trailer format.
|
||||
|
||||
# Resources
|
||||
|
||||
https://semver.org/
|
||||
https://www.conventionalcommits.org/en/v1.0.0/
|
||||
@@ -1,241 +0,0 @@
|
||||
+++
|
||||
title = "Stateful Callbacks in Python"
|
||||
description = "Stateful callbacks in Python."
|
||||
date = 2017-07-10T07:54:00-08:00
|
||||
[taxonomies]
|
||||
tags = ["Python", "callbacks"]
|
||||
+++
|
||||
If you're unfamiliar with what a callback is, don't worry, we can sort that out
|
||||
quickly. If callbacks are old hat for you you might want to skip to
|
||||
[the interesting bit](#stateful-callbacks).
|
||||
|
||||
Simply put, a callback is a function that is passed as an argument to
|
||||
another function which may execute it.
|
||||
|
||||
<!-- more -->
|
||||
|
||||
Take, for example, these functions:
|
||||
|
||||
```python
|
||||
def bar():
|
||||
return "I'm the callback!"
|
||||
|
||||
def foo(func):
|
||||
return func()
|
||||
```
|
||||
|
||||
If we call `foo` like this
|
||||
|
||||
```python
|
||||
>>> foo(bar)
|
||||
"I'm the callback!"
|
||||
```
|
||||
|
||||
then `bar` is a callback.
|
||||
|
||||
## Why Should I Use A Callback?
|
||||
|
||||
There are many reasons to use callbacks. For me, the most compelling is
|
||||
customization. Let's take a look at a Python built-in as an example. Say we
|
||||
have a list of users as dictionaries with a `name` and an `age`:
|
||||
|
||||
```python
|
||||
users = [
|
||||
dict(age=77, name='John Cleese'),
|
||||
dict(age=74, name='Eric Idle'),
|
||||
]
|
||||
```
|
||||
|
||||
Imagine that we want to sort our users. If we had just a list of ages or a list
|
||||
of names we could easily do this with the built-in `sorted` function, but by
|
||||
default Python has no idea how to compare our dictionaries during sorting.
|
||||
|
||||
```python
|
||||
>>> sorted(users)
|
||||
Traceback (most recent call last):
|
||||
File "<stdin>", line 1, in <module>
|
||||
TypeError: unorderable types: dict() < dict()
|
||||
```
|
||||
|
||||
Should it sort by `age`? By `name`? We need to tell Python how this should be
|
||||
done. Fortunately Python provides and the `sorted` function has a keyword
|
||||
argument called `key` that takes, you guessed it, a callback. Let's create some
|
||||
of our own!
|
||||
|
||||
```python
|
||||
def by_age(user):
|
||||
return user['age']
|
||||
|
||||
def by_name(user):
|
||||
return user['name']
|
||||
```
|
||||
|
||||
Armed with these callbacks we can sort our users.
|
||||
|
||||
```python
|
||||
>>> sorted(users, key=by_age)
|
||||
[{'age': 74, 'name': 'Michael Palin'}, {'age': 77, 'name': 'John Cleese'}]
|
||||
>>> sorted(users, key=by_name)
|
||||
[{'age': 77, 'name': 'John Cleese'}, {'age': 74, 'name': 'Michael Palin'}]
|
||||
```
|
||||
|
||||
Since the `sorted` function takes a callback for the `key` argument we are free
|
||||
to customize its behavior. All we have to do is define a function that returns
|
||||
the key we intend to sort by and as long as that's an orderable type Python
|
||||
will take care of the rest.
|
||||
|
||||
## What Does It Mean to Have State?
|
||||
|
||||
So, by now we have something of an idea of what callbacks are, how we can use
|
||||
them, and why, but what's the point of state? State is most easily described as
|
||||
a memory of prior events. This is the core of what every program does and we
|
||||
use it all the time, even if we don't realize it. Heck, even saving a variable
|
||||
involves keeping track of state.
|
||||
|
||||
```python
|
||||
>>> baz = 1 # The Python interpreter is now tracking the state of 'baz'.
|
||||
>>> print(baz) # We can recall that state at a later point.
|
||||
1
|
||||
```
|
||||
|
||||
Basically, we need state if we care to remember what happened previously so
|
||||
that we can make decisions about what to do next.
|
||||
|
||||
## What Normally Happens to State Inside a Callback?
|
||||
|
||||
In our first callback function we didn't define any names. To demonstrate what
|
||||
typically happens to state inside the scope of a callback let's make a function
|
||||
that creates some state.
|
||||
|
||||
```python
|
||||
def quux():
|
||||
plugh = "xyzzy"
|
||||
return plugh
|
||||
```
|
||||
|
||||
When we execute this function we get the expected result.
|
||||
|
||||
```python
|
||||
>>> quux()
|
||||
'xyzzy'
|
||||
```
|
||||
|
||||
After the function is executed we can see that the `plugh` name is not defined.
|
||||
|
||||
```python
|
||||
>>> plugh
|
||||
Traceback (most recent call last):
|
||||
File "<stdin>", line 1, in <module>
|
||||
NameError: name 'plugh' is not defined
|
||||
```
|
||||
|
||||
This is because when the function is finished executing its frame is removed
|
||||
from the [call stack][call stack] along with any locally defined variables. By
|
||||
itself our callback can't remember anything.
|
||||
|
||||
## Stateful Callbacks
|
||||
|
||||
Alright, so we know what callbacks are, we know what state is. How can we
|
||||
combine the two to make a callback that retains its state? As we saw above we
|
||||
can't rely on any state that we define inside our callback. The trick to making
|
||||
a stateful callback is to rely on names bound to an external scope.
|
||||
|
||||
To motivate creating a stateful callback let's say that we still want to sort
|
||||
users like we did above, only now we have 1 Million users. It's going to take
|
||||
a while to sort those users, so it would be nice to have a progress report so
|
||||
we know something is still happening, maybe once per 10,000 users.
|
||||
|
||||
### Using Functions
|
||||
|
||||
To use names bound to an external scope with a plain ol' function as our
|
||||
callback we'll need to take advantage of [closures][closures] (which could be
|
||||
an entirely separate post). Here's a function that allows us to use our
|
||||
original `by_age` and `by_name` sorters while still giving us progress.
|
||||
|
||||
```python
|
||||
def sort_reporter(func):
|
||||
state = dict(count=0) # We can't just call this 'count'...
|
||||
|
||||
def _sort_reporter(user):
|
||||
state['count'] += 1 # Because we'd get an UnboundLocalError here.
|
||||
if state['count'] % 10000 == 0:
|
||||
print("Sorted {count} users.".format(count=state['count']))
|
||||
return func(user)
|
||||
|
||||
return _sort_reporter
|
||||
```
|
||||
|
||||
We can use it like so.
|
||||
|
||||
```python
|
||||
>>> sorted_users = sorted(users, key=sort_reporter(by_name))
|
||||
Sorted 10000 users.
|
||||
Sorted 20000 users.
|
||||
# Lots more of this...
|
||||
```
|
||||
|
||||
How does it work? The key is the `state` dictionary. It lets us keep a mutable
|
||||
reference to a name defined outside the scope of the actual reporter function,
|
||||
`_sort_reporter`. As the `sorted` built-in is processing our `users` each new
|
||||
call to `_sort_reporter` still gets to refer to the original `state`.
|
||||
|
||||
<small><b>Note</b>: We could avoid having a `state` dictionary by using Python
|
||||
3's `nonlocal` keyword, but then I'd miss an opportunity in the [bonus](#bonus)
|
||||
section.</small>
|
||||
|
||||
### Using Classes
|
||||
|
||||
If the functional approach doesn't suit you we can also tackle this problem
|
||||
from an object-oriented angle. Python lets classes define a
|
||||
[`__call__` method][__call__] which makes them callable. This isn't strictly
|
||||
necessary for an OOP approach, but when we're making callbacks it's nice to be
|
||||
able to treat our instances as functions.
|
||||
|
||||
```python
|
||||
class SortReporter:
|
||||
def __init__(self, func):
|
||||
self.func = func
|
||||
self.count = 0
|
||||
|
||||
def __call__(self, user):
|
||||
self.count += 1
|
||||
if self.count % 10000 == 0:
|
||||
print("Sorted {count} users.".format(count=self.count))
|
||||
return self.func(user)
|
||||
```
|
||||
|
||||
Just as easy to use as our functional option.
|
||||
|
||||
```python
|
||||
>>> sorted_users = sorted(users, key=SortReporter(by_age))
|
||||
Sorted 10000 users.
|
||||
Sorted 20000 users.
|
||||
# One eternity later...
|
||||
```
|
||||
|
||||
Conceptually, this works for much the same reason that the functional approach
|
||||
does. The `SortReporter` instance and all its associated state lives on because
|
||||
the `sorted` built-in is carrying around a reference to it and it just pretends
|
||||
to be a plain ol' function whenever `sorted` needs it to be one.
|
||||
|
||||
### Which Should I Use?
|
||||
|
||||
Neither approach is any more or less valid than the other. For this particular
|
||||
example there isn't much more code or complexity either way. I generally regard
|
||||
functions as being simpler than classes, so I prefer those when possible, but
|
||||
classes also provide good structure for more complex callbacks. Try them both!
|
||||
|
||||
#### Bonus
|
||||
|
||||
As <s>homework</s> a bonus, try instantiating a `SortReporter` and examining
|
||||
its `__dict__` attribute. Meditate on what you find there and how it relates
|
||||
to the `state` dictionary in the functional approach.
|
||||
|
||||
If you get really bold and want to try for extra credit assign the return value
|
||||
of the `sort_reporter` function to some variable and examine its `__closure__`
|
||||
attribute. This may help you explain why the `state` dictionary doesn't
|
||||
disappear after the `sort_reporter` function is called.
|
||||
|
||||
[call stack]: https://en.wikipedia.org/wiki/Call_stack
|
||||
[closures]: https://en.wikipedia.org/wiki/Closure_(computer_programming)
|
||||
[__call__]: https://docs.python.org/3/reference/datamodel.html#object.__call__
|
||||
@@ -0,0 +1,122 @@
|
||||
+++
|
||||
title= "Subresource Integrity"
|
||||
description = "Understanding Subresource Integrity"
|
||||
date= 2023-04-29
|
||||
author = "Ian Andwati"
|
||||
[taxonomies]
|
||||
tags= ["subresource integrity", "csp", "cors"]
|
||||
+++
|
||||
|
||||
Understanding Subresource Integrity
|
||||
|
||||
<!-- more -->
|
||||
|
||||
In my junior year while I was learning HTML and CSS I got to use the Bootstrap css library.The easiest way to obtain the files was from a Content Delivery Network(CDN). I could see links such as:
|
||||
|
||||
```html
|
||||
<link
|
||||
href="https://cdn.jsdelivr.net/npm/[email protected]/dist/css/bootstrap.min.css"
|
||||
rel="stylesheet"
|
||||
integrity="sha384-KK94CHFLLe+nY2dmCWGMq91rCGa5gtU4mk92HdvYe+M/SXH301p5ILy+dN9+nJOZ"
|
||||
crossorigin="anonymous"
|
||||
/>
|
||||
```
|
||||
|
||||
I didn't bother with the `integrity` attribute of the link(guess I wasn't curios enough). The integrity attribute has a cryptographic hash value that is important in CDNs and its called the Subresource Integrity.
|
||||
|
||||
Subresource Integrity (SRI) is a security feature in web browsers that helps prevent malicious or compromised scripts from being loaded and executed on a website.a security feature that enables browsers to verify that resources they fetch (for example, from a CDN) are delivered without unexpected manipulation. It works by allowing you to provide a cryptographic hash that a fetched resource must match.
|
||||
|
||||
When a resource, such as a script or stylesheet, fails the SRI check, the browser will block the resource from being loaded and displayed on the website.
|
||||
|
||||
# How Subresource Integrity Helps
|
||||
|
||||
Subresource Integrity (SRI) is a security feature that helps protect against certain types of attacks, such as those that involve malicious modifications to third-party resources that a website loads.
|
||||
|
||||
SRI works by allowing a website to specify a cryptographic hash for a particular external resource (such as a JavaScript file or a CSS stylesheet) that it expects to load. When a user's browser fetches that resource, it checks the hash against the actual contents of the resource. If the hash doesn't match, the browser will refuse to load the resource, indicating that it may have been tampered with.
|
||||
|
||||
This can help prevent a variety of attacks, such as those involving a compromised content delivery network (CDN) or a man-in-the-middle attacker who intercepts and modifies network traffic. By using SRI, a website can help ensure that it is loading only the resources it expects, and that those resources have not been tampered with in transit.
|
||||
|
||||
# Using it
|
||||
|
||||
Using Subresource Integrity (SRI) involves a few steps, but it is relatively straightforward.An integrity value begins with at least one string, with each string including a prefix indicating a particular hash algorithm (currently the allowed prefixes are sha256, sha384, and sha512), followed by a dash, and ending with the actual base64-encoded hash.
|
||||
|
||||
- Generate a cryptographic hash(a base64-encoded cryptographic hash) of the external resource you want to load on your website. This can be done using a hash function such as SHA-256.
|
||||
|
||||
- Add the hash as an attribute to the tag that loads the external resource. For example, for a JavaScript file, you would add the integrity attribute to the `<script>` tag like this:
|
||||
|
||||
```html
|
||||
<script
|
||||
src="https://example.com/example.js"
|
||||
integrity="sha256-ABC123XYZ456"
|
||||
></script>
|
||||
```
|
||||
|
||||
> An integrity value may contain multiple hashes separated by whitespace. A resource will be loaded if it matches one of those hashes.
|
||||
|
||||
Make sure the hash you've generated matches the actual contents of the external resource.
|
||||
|
||||
Test your website to ensure that the external resource is loading correctly and that the SRI feature is working as intended.
|
||||
|
||||
# Tools for generating SRI hashes
|
||||
|
||||
You can use a tool such as the [SRI Hash Generator](https://www.srihash.org/) to generate the hash and check it against the file's contents.
|
||||
|
||||
You can generate SRI hashes from the command-line with openssl using a command invocation such as:
|
||||
|
||||
```sh
|
||||
cat FILENAME.py | openssl dgst -sha384 -binary | openssl base64 -A
|
||||
```
|
||||
|
||||
or with shasum using a command invocation such as:
|
||||
|
||||
```sh
|
||||
shasum -b -a 384 FILENAME.js | awk '{ print $1 }' | xxd -r -p | base64
|
||||
```
|
||||
|
||||
> The pipe-through xxd step takes the hexadecimal output from shasum and converts it to binary.
|
||||
> The pipe-through awk step is necessary because shasum will pass the hashed filename in its output to xxd. That can have disastrous consequences if the filename happens to have valid hex characters in it — because xxd will also decode that and pass it to base64.
|
||||
|
||||
In a Windows environment, you can create a tool for generating SRI hashes with the following code:
|
||||
|
||||
```bat
|
||||
@echo off
|
||||
set bits=384
|
||||
openssl dgst -sha%bits% -binary %1% | openssl base64 -A > tmp
|
||||
set /p a= < tmp
|
||||
del tmp
|
||||
echo sha%bits%-%a%
|
||||
pause
|
||||
```
|
||||
|
||||
To use that code:
|
||||
|
||||
1. Save that code in a file named sri-hash.bat in the Windows SendTo folder in your environment (for example, C:\Users\USER\AppData\Roaming\Microsoft\Windows\SendTo).
|
||||
1. Right-click a file in the File Explorer, select Send to…, and then select sri-hash. You will see the integrity value in a command box.
|
||||
1. Select the integrity value and right-click to copy it to the Clipboard.
|
||||
1. Press any key to dismiss the command box
|
||||
|
||||
# Cross-Origin Resource Sharing and Subresource Integrity
|
||||
|
||||
When verifying the subresource integrity of a resource that originates from a different domain than the document it's embedded in, web browsers will check the resource using Cross-Origin Resource Sharing (CORS) to ensure that the domain serving the resource allows it to be shared with the requesting domain.
|
||||
As a result, the resource must be served with an Access-Control-Allow-Origin header that permits the resource to be shared with the requesting domain. For example, the header could include a wildcard (\*) to allow any domain to access the resource.
|
||||
|
||||
```
|
||||
Access-Control-Allow-Origin: *
|
||||
```
|
||||
|
||||
# How browsers handle Subresource Integrity
|
||||
|
||||
Web browsers employ several mechanisms to handle Subresource Integrity (SRI). When a **script** or **link** element with an integrity attribute is encountered by the browser, it compares the script or stylesheet to the expected hash given in the integrity value before executing the script or applying the stylesheet.
|
||||
|
||||
For resources served from a different domain than the document in which it's embedded, the browser also checks the resource using Cross-Origin Resource Sharing (CORS) to ensure that the origin serving the resource allows it to be shared with the requesting domain. This additional check is important for preventing unauthorized access to sensitive resources.
|
||||
|
||||
If the script or stylesheet fails to match its associated integrity value, the browser will refuse to execute the script or apply the stylesheet and return a network error, indicating that the fetching of that script or stylesheet failed. This behavior ensures that the resource being loaded is trustworthy and helps to protect users from potential security threats.
|
||||
|
||||
# References
|
||||
|
||||
- [https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP](https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP)
|
||||
- [https://www.srihash.org/](https://www.srihash.org/)
|
||||
- [https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity)
|
||||
- [https://html.spec.whatwg.org/multipage/semantics.html#attr-script-integrity](https://html.spec.whatwg.org/multipage/semantics.html#attr-script-integrity)
|
||||
- [https://html.spec.whatwg.org/multipage/semantics.html#attr-link-integrity](https://html.spec.whatwg.org/multipage/semantics.html#attr-link-integrity)
|
||||
- [https://w3c.github.io/webappsec-subresource-integrity/#the-integrity-attribute](https://w3c.github.io/webappsec-subresource-integrity/#the-integrity-attribute)
|
||||
@@ -0,0 +1,67 @@
|
||||
+++
|
||||
title = "What say you?"
|
||||
description = "Adding a comment system by utterances"
|
||||
author = "Ian Andwati"
|
||||
date = 2023-08-24T15:34:14.160Z
|
||||
[taxonomies]
|
||||
tags = ["blog", "utterances"]
|
||||
+++
|
||||
|
||||
Free speech enhanced
|
||||
|
||||
<!-- more -->
|
||||
|
||||
There is no point in writing online if your readers cannot provide feedback for your writings. You might as well be shouting at a void. My ideal commenting system has to be free, open source and should not be a hassle to setup. Luckily, [utterances 🔮](https://github.com/utterance/utterances) checks all of these conditions
|
||||
|
||||
It is a lightweight comments widget built on GitHub issues. Use GitHub issues for blog comments, wiki pages and more! The notable features are:
|
||||
|
||||
- Open source. 🙌
|
||||
- No tracking, no ads, always free. 📡🚫
|
||||
- No lock-in. All data stored in GitHub issues. 🔓
|
||||
- Styled with Primer, the css toolkit that powers GitHub. 💅
|
||||
- Dark theme. 🌘
|
||||
- Lightweight. Vanilla TypeScript. No font downloads, JavaScript frameworks or polyfills for evergreen browsers. 🐦🌲
|
||||
|
||||
## How it works
|
||||
|
||||
When Utterances loads, the GitHub [issue search API](https://developer.github.com/v3/search/#search-issues) is used to find the issue associated with the page based on url, pathname or title. If it cannot find an issue that matches the page, [utterances-bot](https://github.com/utterances-bot) will automatically create an issue the first time someone comments.
|
||||
|
||||
To comment, users must authorize the utterances app to post on their behalf using the GitHub [OAuth flow](https://developer.github.com/v3/oauth/#web-application-flow). Alternatively, users can comment on the GitHub issue directly.
|
||||
|
||||
# Setting up Utterances
|
||||
|
||||
### Repository
|
||||
|
||||
Choose the repository utterances will connect to.
|
||||
|
||||
Make sure the repo is public, otherwise your readers will not be able to view the issues/comments.
|
||||
Make sure the utterances app is installed on the repo, otherwise users will not be able to post comments.
|
||||
If your repo is a fork, navigate to its settings tab and confirm the issues feature is turned on.
|
||||
|
||||
```
|
||||
repo:
|
||||
username/repo
|
||||
```
|
||||
|
||||
A public GitHub repository. This is where the blog post issues and issue-comments will be posted.
|
||||
|
||||
### Theme
|
||||
|
||||
Choose an Utterances theme that matches your blog.
|
||||
|
||||
### Enable Utterances
|
||||
|
||||
Add the following script tag to your blog's template. Position it where you want the comments to appear. Customize the layout using the .utterances and .utterances-frame selectors.
|
||||
|
||||
```js
|
||||
<script
|
||||
src="https://utteranc.es/client.js"
|
||||
repo="[ENTER REPO HERE]"
|
||||
issue-term="title"
|
||||
theme="gruvbox-dark"
|
||||
crossorigin="anonymous"
|
||||
async
|
||||
></script>
|
||||
```
|
||||
|
||||
The complete documentation can be found at [https://utteranc.es](https://utteranc.es/)
|
||||
@@ -0,0 +1,54 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>50x Borked</title>
|
||||
<meta name="author" content="Reilly Tucker Siemens">
|
||||
<meta name="description" content="50x Borked">
|
||||
<meta property="og:site_name" content="tuckersiemens.com">
|
||||
<meta property="og:locale" content="en-us">
|
||||
<meta property="og:title" content="50x Borked">
|
||||
<meta property="og:description" content="50x Borked">
|
||||
<meta property="og:image" content="https://tuckersiemens.com/self.jpg">
|
||||
<meta property="twitter:card" content="summary">
|
||||
<meta property="twitter:site" content="@TuckerSiemens">
|
||||
<meta property="twitter:title" content="50x Borked">
|
||||
<meta property="twitter:description" content="50x Borked">
|
||||
<meta property="twitter:image" content="https://tuckersiemens.com/self.jpg">
|
||||
<link rel="dns-prefetch" href="https://fonts.googleapis.com/">
|
||||
<link rel="stylesheet" href="/site.css">
|
||||
<link rel="icon" href="/favicon.svg" type="image/svg+xml">
|
||||
</head>
|
||||
<body>
|
||||
<main id="content">
|
||||
<header>
|
||||
<h1>50[0234] Borked</h1>
|
||||
</header>
|
||||
<figure>
|
||||
<video src="50x.mp4" type="video/mp4" autoplay loop controls></video>
|
||||
<figcaption>
|
||||
Video from <a href="https://www.sakugabooru.com/post/show/174648">sakugabooru</a>
|
||||
</figcaption>
|
||||
</figure>
|
||||
</main>
|
||||
<footer>
|
||||
<nav>
|
||||
<a href="/">Home</a>
|
||||
⬡
|
||||
<a href="/posts">Posts</a>
|
||||
⬡
|
||||
<a href="/tags">Tags</a>
|
||||
⬡
|
||||
<a href="/contact">Contact</a>
|
||||
</nav>
|
||||
<p>
|
||||
©
|
||||
2021 Reilly Tucker Siemens ⬡
|
||||
<a href="https://github.com/reillysiemens/tuckersiemens.com">Source</a>
|
||||
⬡
|
||||
<a href="/rss.xml">Feed</a>
|
||||
</p>
|
||||
</footer>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,3 @@
|
||||
<svg height="128" width="128" xmlns="http://www.w3.org/2000/svg">
|
||||
<polygon points="64 4, 124 32, 124 96, 64 124, 4 96, 4 32" fill="#b16286"/>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 151 B |
|
After Width: | Height: | Size: 103 KiB |
@@ -0,0 +1 @@
|
||||
@import url("https://fonts.googleapis.com/css?family=Noto+Sans|Noto+Serif|Noto+Sans+Mono");:root{--default-serif-font: "Noto Serif", serif;--default-sans-font: "Noto Sans", sans-serif;--default-mono-font: "Noto Sans Mono", monospace;--default-font-size: 16px;--header-font-size: 2.0em;--default-line-height: 1.5;--background-color: #1d2021;--foreground-color: #ebdbb2;--link-color: #458588;--muted-color: #928374;--primary-accent-color: #b16286;--secondary-accent-color: #689d6a;--code-background-color: #282828;--inline-code-background-color: #3c3836}.accent{color:var(--primary-accent-color)}html,body{height:100%}body{display:flex;flex-direction:column;margin:0;line-height:var(--default-line-height);font-size:var(--default-font-size);font-family:var(--default-serif-font);background-color:var(--background-color);color:var(--foreground-color);border-top:.5em solid var(--primary-accent-color)}main{flex:1 0 auto;}#content{max-width:48em;margin-left:auto;margin-right:auto;padding-right:1.4em;padding-left:1.4em;padding-bottom:1.4em}header{overflow:hidden;margin-top:1.4em;margin-bottom:1.4em;margin-left:1em;margin-right:1em}header>h1{text-align:center;font-size:var(--header-font-size);font-family:var(--default-sans-font);padding-top:2.8rem;margin-top:0;margin-bottom:.7rem}header .post-meta{text-align:center;padding-top:.5em;padding-bottom:.9em;margin-bottom:0}.post-meta time{color:var(--muted-color)}h2,h3,h4,h5,h6{padding-top:1.2rem;margin-bottom:.2rem}p{padding-top:.4em;margin-bottom:1em}blockquote{font-family:var(--default-sans-font);font-style:italic;margin:0 1.25rem;padding-left:1.25rem;padding-right:1.25rem;border-left:3px solid var(--secondary-accent-color)}a{color:var(--link-color);text-decoration:none}:where(h2,h3,h4,h5,h6)>a[href^="#"]::before{content:"#";display:inline-block;width:1.2rem;margin-left:-1.2rem;opacity:0}:where(h2,h3,h4,h5,h6):hover>a[href^="#"]::before{opacity:1}:where(h2,h3,h4,h5,h6):target>a[href^="#"]::before{opacity:1;color:var(--primary-accent-color)}img,video{width:100%;height:auto}figure{margin:0}figcaption{font-size:.85rem;color:var(--muted-color)}pre{display:block;padding:.25rem 1.25rem;font-size:85%;border-left:.25rem solid var(--primary-accent-color)}code{padding:0 .4rem;background-color:var(--inline-code-background-color);border-radius:2px}pre,code{overflow:auto;font-family:var(--default-mono-font)}pre>code{padding:0;background-color:var(--code-background-color);border-radius:unset}table{display:block;overflow-x:auto;margin-left:auto;margin-right:auto;text-align:left;border-collapse:collapse}th[scope=col]{padding-top:.4em;border-bottom:3px solid var(--secondary-accent-color)}th[scope=row]{border-right:3px solid var(--secondary-accent-color)}td{text-align:center;border:1px solid var(--muted-color)}th,td{vertical-align:bottom;padding-left:.4em;padding-right:.4em;border:1px solid var(--muted-color)}footer{border-top:.5em solid var(--primary-accent-color);text-align:center;background-color:var(--code-background-color);padding:1.4em;flex-shrink:0;}footer a{color:var(--secondary-accent-color)}footer>nav>a{margin:0 .4em}footer p{padding-top:0}footer p,footer a{font-family:var(--default-sans-font);font-variant-caps:all-small-caps}@media (min-width: 470px){:root{--default-font-size: 17px}header{margin-left:2.8em;margin-right:2.8em}}@media (min-width: 625px){:root{--default-font-size: 18px}}@media (min-width: 802px){:root{--default-font-size: 19px}}@media (min-width: 1003px){:root{--default-font-size: 20px}}@media (min-width: 1225px){:root{--default-font-size: 21px}:where(h2,h3,h4,h5,h6)>a[href^="#"]::before{width:1.8rem;margin-left:-1.8rem}}@media (min-width: 1496px){:root{--default-font-size: 22px}}
|
||||
@@ -1,82 +0,0 @@
|
||||
==================================================================
|
||||
https://keybase.io/reillysiemens
|
||||
--------------------------------------------------------------------
|
||||
|
||||
I hereby claim:
|
||||
|
||||
* I am an admin of https://tuckersiemens.com
|
||||
* I am reillysiemens (https://keybase.io/reillysiemens) on keybase.
|
||||
* I have a public key with fingerprint EA26 C944 1A6C 40E4 0E73 B4EC 1B45 8E7C 7CB1 54A9
|
||||
|
||||
To do so, I am signing this object:
|
||||
|
||||
{
|
||||
"body": {
|
||||
"client": {
|
||||
"name": "keybase.io node.js client",
|
||||
"version": "0.7.5"
|
||||
},
|
||||
"key": {
|
||||
"fingerprint": "ea26c9441a6c40e40e73b4ec1b458e7c7cb154a9",
|
||||
"host": "keybase.io",
|
||||
"key_id": "1B458E7C7CB154A9",
|
||||
"uid": "eb4bae03913d61f8dbaecec09f80f300",
|
||||
"username": "reillysiemens"
|
||||
},
|
||||
"merkle_root": {
|
||||
"ctime": 1429866644,
|
||||
"hash": "e0d63404eea56d53642bc5843899d777ab377056b09c5e7bb71264f686bed4ebf84e74a7a8eb3bcce875a736563ddb16b8dc8e64407c1faf37954a0e44a29570",
|
||||
"seqno": 209926
|
||||
},
|
||||
"service": {
|
||||
"hostname": "tuckersiemens.com",
|
||||
"protocol": "https:"
|
||||
},
|
||||
"type": "web_service_binding",
|
||||
"version": 1
|
||||
},
|
||||
"ctime": 1429866655,
|
||||
"expire_in": 157680000,
|
||||
"prev": "2fc94aa693b2d1a220a6dfe6d250809aab5495ba850356e72d03998d8763e716",
|
||||
"seqno": 39,
|
||||
"tag": "signature"
|
||||
}
|
||||
|
||||
which yields the signature:
|
||||
|
||||
-----BEGIN PGP MESSAGE-----
|
||||
Version: GnuPG v2
|
||||
|
||||
owFdkntQFVUcxwHTkAli5CGJpm1SqQzu8+zuDZoAHQokwGCkgLmes3suLI+717v3
|
||||
8hBuNJSOxYwCyjNEM0EaJwswJrMZ5BED0oMgEsTBGXVUBGdUfDQ5UrsMTY8z+8c5
|
||||
ez6/7+/3/f1OhfcSNw/34K37S0rakj93H5pGbikmzyPFBFLlIsJUTEi5CrY6jJ0V
|
||||
5mHCROTgIgQ1HKao66yqjMOytXWLTCiRj+2aolp1igzjwzjCFWrgRrBFsWZiu82u
|
||||
GFoEhjSQRJalIJBYEusfzyAWSxRiOQHzEi8himOhqEtmqZrjP1mJBU2zIut/qSid
|
||||
38pH89FROh9p8M6FC4xYBDHJiBQjA8oiyPpJwhIpWgTSwpCkAWrYvmjJjpXc3CJN
|
||||
wXnYqhlF52F7Ti4221V1wbnkUAyOYmlRAACwrF4W1LKMPKQMGJZkMYYckDkGsDSS
|
||||
OIFlBFGUeZ6HiOF5kgOIFCUO8wjxFA1YCxAAwjKLkUVgMc9CHgoYMUiSsMBzkGcA
|
||||
BxhZRhRAgiwJWM9I8hJlgRaGF/W+6B1jIS1yvOFDw7usKmGiSVGkgcs42/MVCRt1
|
||||
G71btOhwSjnGcBYshklqnh5ps6sOVVJz9essh8OmmQzrZxxFNiOgACPzopQZKVZZ
|
||||
n9+/B0zp6P/awnGhBC60KXZsVgyC44FA6stIhPN1SdqizxxCIDKIlilI0yQEsgUD
|
||||
meZIgRQhRBwrcggKHMlwAPO0rA9QFGSBBwzmKfCPWUYMJRwwU9fUlEwrdDjtmHB9
|
||||
5LH6KTd3D7dlSz2MJ+zmtdz373c9ctTX7XjvsaiJMVS1q6JPLpkJeHK8o6g9/7U/
|
||||
fc5eS2xRAiK+OPhZGrFFVR6/DjegNd53/5iLVHePdAWP1JMDE/1Nxzbv/bY5zhZI
|
||||
zFb7ppcWSVeeHXQNQ1fjzHB5Z5LPvam15fs/LM7a26V1HZxMfyGjNPz9edT6/TPj
|
||||
c/FPKpNnU6PiY+JP3a1Ju1lxO2zl7QDbxd9PvHLO78seT+W5lx9S2RkhiaaX3pge
|
||||
W4LO91zwKqu49vb0YPV41cTHcUmxsTEtLS9qDW2zYw8P3DsyKp+ZTH++vKB228z2
|
||||
5DeTuiamX3XUXf457daB6k/mmh94Td2vs9TFlx4+nXD7pHAxpj4R3v1uvjf1xLuX
|
||||
BzZvd86UtkzdTIisT63daC+81DlaadoZiOev7gv64Zss6kLNjbLrV5ed7/QJVHz9
|
||||
Sk5zO/qTen+LWNUY51yRMThX3QdDhwtii9lNYlXkoeD+X8fvfRW4vpyt+6W0z/RB
|
||||
fnx0dPfaB0ebGvbYUobYMk9tU3jXeEVffPudiMOtA+tvmMse/RjkvXRVlEpGt7ra
|
||||
x84OZdf03hA8rKk/3W8FY9u6H0+OiiFPX38nZLA3ZnvT11GpG9+64+8XvK9yx60N
|
||||
59jKFPHWxCkqIbJxdf8W1/KV7o4VHbUh4WsCOm5Wzp1sejQ6fOm9+aQrzXt27h08
|
||||
9GlDm9O9LNPfGRPUbYYm/xO7ewqL/wI=
|
||||
=JQ/H
|
||||
-----END PGP MESSAGE-----
|
||||
|
||||
And finally, I am proving ownership of this host by posting or
|
||||
appending to this document.
|
||||
|
||||
View my publicly-auditable identity here: https://keybase.io/reillysiemens
|
||||
|
||||
==================================================================
|
||||