Gergely Polonkai - blog/2022-06-18T03:02:00+00:00Keyboardio’s Model 100 keyboard – a review2022-06-18T03:02:00+00:002022-06-18T03:02:00+00:00Gergely Polonkaitag:None,2022-06-18:blog/keyboardio-model100-keyboard-review/<p><span class="caps">TL</span>;<span class="caps">DR</span>: despite the fact that i almost have to re-learn typing, this thing is awesome and i think
everyone typing a decent amount of letters a day should get a split, or even a split+vertical keyboard.</p>
<hr class="docutils" />
<p>I’m not a keyboard enthusiast, but i heard so much about …</p><p><span class="caps">TL</span>;<span class="caps">DR</span>: despite the fact that i almost have to re-learn typing, this thing is awesome and i think
everyone typing a decent amount of letters a day should get a split, or even a split+vertical keyboard.</p>
<hr class="docutils" />
<p>I’m not a keyboard enthusiast, but i heard so much about split keyboards and <a class="reference external" href="https://shop.keyboard.io">Keyboardio</a>’s Model 01 before that i decided to back their <a class="reference external" href="https://www.kickstarter.com/projects/keyboardio/model-100">Model 100 on
Kickstarter</a> about a year ago. It’s
super expensive in my book (about 20% of my monthly salary when i ordered it) but since they
announced the project way before the <span class="caps">KS</span> campaign i had time to slowly put aside some money every
month. This makes it the first electronic device that i actually wanted so much (and probably
needed; ergonomics is important, as i learned the hard way coming closer to 40) that i put aside
money for it. Their hunt-and-pecking chicken in the promotion video was also pretty convincing.</p>
<p>We got a lot of info on what is happening during the time between backing and delivery.
Some video footage of how the keyboards are made, from production to <span class="caps">QA</span>, was one of them, and it
looked really awesome. Even though i only gave my money, the whole project felt more personal
this way.</p>
<p>They had a lot of delay in production due to various reasons (<span class="caps">COVID</span> and Chinese new year being the
biggest among them) but when i finally got a mail that said</p>
<blockquote>
your early-delivery Model 100 is due to be ready to ship out this week. I’m writing today to
confirm your shipping address. We’ll send you another email with tracking information as soon as
your keyboard ships out.</blockquote>
<p>i was super excited about it and couldn’t wait for it to arrive.</p>
<p>Shipping was handled by FedEx; they did duty handling and delivery quick and with great care.
This post is not their review, but they surely deserve a shout-out. My package even arrived one
day before they originally promised.</p>
<p>And now, finally onto the review itself! Keep in mind that things below are not in order of
importance but more like in the order of how i discovered them.</p>
<div class="section" id="the-package">
<h2>The package</h2>
<p>I really don’t like unboxing videos; i just find them boring. For this keyboard i tried to make
one, but quickly dumped the idea. Then i tried to make pictures of the steps, but still no luck:
i got two photos of the early steps (opening the first box) than i gave up. So if you are here
for unboxing material, you should close the page now.</p>
<p>The package contained two black boxes. They look really cool, with the Keyboardio logo on them.
The smaller box contains the stands (which have a standard tripod screw, so you can technically
put your keyboard halves on camera tripods next to your desk, if that’s your game). The bigger
box contained nothing but a travelling case. It’s pretty sturdy, covered with black textile, and
with a bumpy logo. The keyboard has travelled all the way from China in that case; travelling
case redefined.</p>
<p>Inside the case were the two keyboard halves, a <span class="caps">USB</span>-C to <span class="caps">USB</span>-A cable, two <span class="caps">RJ45</span> cables (a short one
when you use the keyboard as one piece, and a long one if you want to use it split), a key switch
puller (because if you don’t like your switches, you can switch to different switches; i’ll show
myself out), two pieces of plastic to hold the two halves together in a linear or a rooftop shape,
and a screwdriver. Because your warranty is not void if you take the keyboard apart. Because why
should it be?</p>
<p>Assembling the keyboard, especially in a split fashion, is not a big deal, just connect one of the
<span class="caps">RJ45</span> cables to both halves, then use the <span class="caps">USB</span>-C cable to connect the left half to your computer,
and you are good to go.</p>
</div>
<div class="section" id="the-look">
<h2>The Look</h2>
<p>This thing has got the look; it is beautiful! The folks at Keyboardio didn’t name it “heirloom
keyboard” for nothing. When i ordered it i had a green desk so i chose the walnut case, which is
a darker shade of brown. Since then i switched to a new desk i got from my brother-in-law, which is
also dark brown (although slightly darker then the keyboard’s enclosure), so i’m probably up to
painting or re-foiling my desk soon. Still, it looks good on my desk, even if it doesn’t
stand out.</p>
<p>My new(ish) desk has a drawer-like thingy for the keyboard, but its height is designed to hold a
traditional keyboard; for the Model 100 on its stands it’s not high enough; plus it’s about 15cm
below the desk itself which is too low for my height. On the desk i already had an <span class="caps">XP</span>-<span class="caps">PEN</span> tablet
so the Model 100 got placed next to it in a split fashion, on both sides. I had to drill two
holes in the desk for the <span class="caps">RJ45</span> cable so it doesn’t take unnecessary space, but it’s a worthy compromise.</p>
<div class="figure">
<img alt="Photo of my desk, a dark brown computer desk with a monitor stand and a separate monitor next to it." src="images/desk-photo.png" />
<p class="caption">This is my current setup: a tablet with wings! And a messy table. And a <a class="reference external" href="https://en.wikipedia.org/wiki/Rubber_duck_debugging">rubber duck</a>. And a rubber shark for tougher
problems. And some retro-computing.</p>
</div>
</div>
<div class="section" id="the-feel">
<h2>The Feel</h2>
<p>With the adjustable stands it’s really comfortable. It took me a few tries, but now that it’s
done my hands never felt this good on a keyboard before. It’s worth noting that i’m typing
extensively on traditional keyboards since my early teens, so my arms developed a little different
(my radius bones are slightly longer than they should be). I didn’t even have those V-shaped,
so-called ergonomic ones, although i yearned for one. Some consequences of this are that my hands
don’t feel distorted at all when i use traditional keyboards, stuff designed to be ergonomic for
the general public might not be comfortable for me immediately, and typing on a split/ortholinear
keyboard is totally new ground for me because for me <em>this</em> is distorted (a bit).</p>
<p>Holding your fingers on this keyboard is slightly different from traditional ones as you don’t
have to keep them on a straight line, but the curve of the home row actually follows the curve of
your fingers. You know those little bumps on the <kbd>F</kbd> and <kbd>J</kbd> keys? In case you don’t
know, they are there to help you position your hands on the “home row” for touch typing (maybe the
reason they call it touch typing instead of blind typing?); you have to find them with your index
fingers, and you’re all set. Now, since you hold your hands a bit different on the Model 100,
they didn’t only add this bump to the usual <kbd>F</kbd> / <kbd>J</kbd> keys, they are also present on
<kbd>A</kbd> and <kbd>;</kbd> so you can easily position your pinkies, too. Really thoughtful design!</p>
<p>Typing on it after using traditional keyboards for all these years is… different, to say the
least. It took me some time to get up to <em>a</em> typing speed (ie. not hunt-and-pecking; heck, typing
my 14 characters long desktop password took 3 tries for the first time) and it was clear that it
should probably take days, if not weeks, to get up to my original speed. It’s a completely
different experience and calls for lots of learning and maybe a layout change, too. For the
record i use a Dvorak inspired layout for several years now; on this keyboard the letters are more
or less where they should be, but some symbols are not that comfortable to reach than they were on
my old one. A lot of non-letter keys are at different places: reaching for the top right for
backspace and top left for escape was an issue for a while, but it’s almost completely gone now.
It’s funny, though, that the keys arranged in this ortholinear fashion didn’t make my head spin
too much. I did reach to the side a bit sometimes, but i got lost of that habit after only two days.</p>
</div>
<div class="section" id="the-learning-process">
<h2>The learning process</h2>
<p>Even though the designers <a class="reference external" href="https://shop.keyboard.io/pages/model100">suggest</a> to start using
split keyboards gradually (start with 15ish minutes in the morning and increase your usage every
day) i used it almost exclusively for my first full workday. We had a short emergency at my $job
for which i used my old keyboard because of speed, but other than that i used the Model 100. What
can i say? I’m a rebel. Also, this method worked really good for me when <a class="reference external" href="blog/2013/3/13/dvorak-and-me.html">i switched to Dvorak</a>, so i guessed it will be fine for split keyboards, too.</p>
<p>Even after a day my typing speed was still far from my old one, but it got better every hour. I
still occasionally pressed Num Lock when i wanted to Backspace (Backspace is under your left
thumb on the default layout) and i often pressed Backspace when i wanted Space (Space is under
your right thumb). Enter being to the left of my right index instead of being to the right of my
right pinky was also confusing sometimes. I probably haven’t looked at my keyboard this much for,
i don’t know… 20 years? But i was slowly getting rid of hunt-and-pecking which was a win in my book.</p>
<p>After more than a week the keyboard felt really good. I was already pretty close to my original
typing speed which wasn’t super fast, but decent enough to amaze some non-techie persons.</p>
</div>
<div class="section" id="technicalities">
<h2>Technicalities</h2>
<p>This section is specific to <a class="reference external" href="https://kaleidoscope.readthedocs.io/">Kaleidoscope</a>, the factory
firmware of the Model 100. If you want to use a different firmware (i guess you could, but don’t
ask me how), just skip ahead.</p>
<p>Early delivery keyboards arrived with <span class="caps">QWERTY</span> keycaps and a default firmware, both holding some
strange things.</p>
<div class="section" id="left-half">
<h3>Left half</h3>
<p>First, there is a <kbd>prog</kbd> key at the top left corner. It is used by the keyboard’s boot
loader to enter programming mode (ie. firmware upgrade), but otherwise it can be used as any other
key (more on that later).</p>
<p>There’s also a <kbd>led</kbd> key, which sounded strange, but who knows… after giving it a few pushes
it turned out that it was switching between different <span class="caps">LED</span> modes, of which there are plenty from a
rainbow wave to dull single colour backlight.</p>
<p><kbd>Esc</kbd> moved to the bottom right of the left half is also strange, but i guess there’s a
reason for that.</p>
<p><kbd>Page Up</kbd> and <kbd>Page Down</kbd> living on the leftmost column was also a big surprise and, as
time showed, a real waste of precious keys for me as a hardly ever use those.</p>
</div>
<div class="section" id="right-half">
<h3>Right half</h3>
<p>The first surprise was that although it <em>does</em> have a logo key (with a Keyboardio butterfly on
it), it doesn’t act as one. Rather than that, it’s actually a right <kbd>Alt</kbd>, <span class="caps">AKA</span> <kbd>AltGr</kbd>
key.</p>
<p>On the top left there is an <kbd>any</kbd> key. Like, a key actually labelled as <tt class="docutils literal">any</tt>. It
sends a random keystroke (showing you the power of macros).</p>
</div>
<div class="section" id="thumb-rows">
<h3>Thumb rows</h3>
<p>Under four rows of keys (except the middle two columns which only have three keys each) are four
“thumb keys”, arranged in a way that your thumb can comfortably reach them.</p>
<p>On the left side, they are <kbd>ctrl</kbd>, <kbd>backspace</kbd>, <kbd>cmd</kbd>, and <kbd>shift</kbd>; on the right side they are <kbd>shift</kbd>, <kbd>alt</kbd>, <kbd>space</kbd>, and <kbd>ctrl</kbd>.</p>
<p>The <kbd>cmd</kbd> key’s label is foreign to <span class="caps">PC</span> people but is used a lot on Macs. Turned out it’s the
<kbd>Super</kbd> key (<span class="caps">AKA</span> logo, <span class="caps">AKA</span> Windows key).</p>
<p>There are also two palm keys labelled as <kbd>fun</kbd>, which are placed so they are easy to press
with your palms, but not easy enough so that you press them all the time. By default they act a
bit like a regular notebook’s <kbd>fn</kbd> keys, adding extra functions to most keys like media
player control, arrow keys, and mouse operations.</p>
</div>
<div class="section" id="layers">
<h3>Layers</h3>
<p>The default firmware has 5 layers: one for the basic keys (what you see printed on the keycaps), a
number pad layer, one for the extra functions (what you get by using the <kbd>fun</kbd> keys), and two
empty ones you can use for your own dark goals (after configuring them in Chrysalys; more about
that later).</p>
<p>Layers don’t have a fixed order but they are stacked when you switch between them. You have a
default layer, and when you press a layer changing key another one is put on top of that. Now
when you press a key and it has a function configured, you get that function. The key can also be
transparent on that layer, so the same key’s function from the layer below (the one previously
active) will be activated. If it’s transparent on that layer, too, then the layer below that will
be taken, etc. If you hit a transparent key on the bottom layer, it acts as a blocked key. Keys
can also be configured as blocked; blocked keys don’t send anything, as if they weren’t even there.</p>
<p>There are four different layer change modes:</p>
<ul class="simple">
<li>Layer Shift is temporary, like how the <kbd>Shift</kbd> key works</li>
<li>Locking also pushes the new layer onto the old one like shifting does, but it remains active,
like how <kbd>Caps Lock</kbd> works</li>
<li>Moving clears the whole stack and makes the target layer the default one. This is like changing
your keyboard layout in the <span class="caps">OS</span></li>
<li>The OneShot plugin (enabled by default) adds another method, labelled “layer shift for next
action” in Chrysalis. If you press such a key once, it will act like a layer lock for the next
single keypress. If you press it twice, it will act as a layer lock, until the OneShot effect
is cancelled (with the <kbd>Escape</kbd> key, or with a preconfigured “OneShot cancel” key).
Pressing it once more will also disable the OneShot effect.</li>
</ul>
</div>
<div class="section" id="mouse-functions">
<h3>Mouse functions</h3>
<p>The keyboard itself doesn’t just advertise itself as a keyboard, but a fully compliant <span class="caps">HID</span>, Human
Interface Device. It can also act as a mouse, except instead of moving around a piece of plastic
on your desk, you press keys. I put away my actual mouse for about 30 minutes and made myself
using the mouse functions exclusively, and i must say it’s not terrible. For the record, i don’t
use the mouse too much; mostly for gaming only. Well, for gaming you probably won’t use a
keyboard based mouse (especially not for mouse-heavy shooters), but for my desktop use case it’s
totally enough.</p>
<p>The most useful feature here (for me) is having a scroll wheel integrated into my keyboard. This
way i don’t have to reach for the mouse when all i want to do is scroll down a bit (for one-page
scrolling there’s still <kbd>Space</kbd>).</p>
<p>Mouse movement is… not ideal. It starts very slowly and gradually increases the pointer speed to
so fast you won’t be able to follow it with your eyes. There’s also mouse warping for moving to
the different corners of your screen, but it doesn’t work well on my multi-monitor Sway setup; i
haven’t tried it on other OSes or desktop environments.</p>
</div>
<div class="section" id="leds">
<h3>LEDs</h3>
<p>I experimented with the LEDs, too. Every key has it’s own <span class="caps">RGB</span> <span class="caps">LED</span> underneath and all of them can
be customised to be lit in different colours. The firmware has a lot of modes from your average
single-colour backlight through moving rainbow gradients to more exotic ones. My favourite is
called Stalker (we agreed with the authors that it’s a pretty bad name) with the blazing trail
effect: whenever you press a key it lights up in white, then fades through red to turned off. It
looks really cool, especially since i have accumulated some actual typing speed so the LEDs don’t
have time to turn off before i press the same key again. Not the ideal one for typing passwords
in public, though. Also, if you use the ColorMap mode, you can configure a different <span class="caps">LED</span> layout
for any or all of your layers which can be super helpful in some cases. Unfortunately the <span class="caps">LED</span>
mode is global, so you can’t have different modes for different layers, unless you write a plugin
(although you can get close enough with colormap).</p>
</div>
</div>
<div class="section" id="the-software">
<h2>The Software</h2>
<p><a class="reference external" href="https://github.com/keyboardio/Chrysalis">Chrysalis</a> is a “desktop application” to configure
your Kaleidoscope based keyboard on the fly. It’s a really good one; my only problem with it is
that it’s based on Electron (ie. not a real desktop app, just a fancy web page running in a
dedicated browser), although i do understand the choice: iterating over new features is really
straightforward and fast this way without risking that one platform falls behind another.</p>
<p>I had some trouble making it run on my desktop (i’m using <a class="reference external" href="https://swaywm.org/">Sway</a>, a Wayland
based window manager; it’s not your everyday Linux setup unless you’re a geek like me), and there
were some permission issues, too; but it all worked out fast thanks to their developer, <a class="reference external" href="https://asylum.madhouse-project.org/about/">@algernon</a>’s help (one of the software’s developers, and with
whom we share first names!)</p>
<p>Chrysalis has just the right amount of bells and whistles and it can fully customise every single
key (given you use the official firmware, which you don’t have to, but i really think you should,
at least until you get used to it). There are some things to grasp before you start using it
(layers being the one of them), but after that one can probably use it without much of a thought.</p>
</div>
<div class="section" id="closing-thoughts">
<h2>Closing thoughts</h2>
<p>I originally wanted to write about my final setup (as of now) but it would have doubled the length
of this post, so it will have its own.</p>
<p>All in all, the Model 100 is a really good keyboard. I could probably have spent this money
better given that i’m a family man with kids and a house that constantly needs fixing, but i’m
happy i didn’t. If you type a lot then you should probably treat your hands with a split and/or
ortholinear keyboard. It gives a fantastic experience.</p>
<p>Also, given the whole thing is open source, and all the parts you need are available to buy you
can even build one at home, but if you have the money you should definitely buy it. If for
nothing else, then as a heirloom.</p>
</div>
Using the MAC address of an ESP8266’s WiFi adapter as the MQTT unique ID2021-01-31T05:35:00+00:002021-01-31T05:35:00+00:00Gergely Polonkaitag:None,2021-01-31:blog/using-esp8266-macaddress-as-mqtt-unique-id/<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">This is a long story. If you are interested only in the solution, <a class="reference internal" href="#solution">click here</a></p>
</div>
<div class="section" id="the-story">
<h2>The story</h2>
<p>I recently dug my nose in the Arduino world, my goal being to build my own weather station. After
experimenting with Arduinos and other boards of different sizes, i finally settled using …</p></div><div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">This is a long story. If you are interested only in the solution, <a class="reference internal" href="#solution">click here</a></p>
</div>
<div class="section" id="the-story">
<h2>The story</h2>
<p>I recently dug my nose in the Arduino world, my goal being to build my own weather station. After
experimenting with Arduinos and other boards of different sizes, i finally settled using <span class="caps">ESP8266</span>
based NodeMCU boards. They are cheap, have a well developed and maintained library for the
Arduino <span class="caps">IDE</span>, and can be accessed via WiFi so no cabling is needed throughout my house.</p>
<p>The first version was based on Prometheus, running on a Raspberry Pi. Prometheus periodically
queried the station for the weather data, put it on a graph, and all was set…</p>
<p>Except keeping a web server constantly up is really power consuming so it can’t be operated from a
battery; i had to install the station near a mains socket, which is in a place in wind shade,
providing less accurate temperature readings. It also takes away a precious wall socket, of which
i have only two outside.</p>
<p>I got a little stuck in an <a class="reference external" href="https://en.wikipedia.org/wiki/XY_problem">X-Y problem</a>, and started
looking for another solution, based on a Prometheus Push Gateway. While browsing documentation
and generally hanging around the Interwebz, i stumbled upon the Home Assistant project (which was
on my ToDo list for a looong time anyway). Without hesitating, i quickly backed up the <span class="caps">SD</span> card of
my Raspberry Pi and installed Hassio on it. I never looked back since.</p>
<p>With Home Assistant installed and configured, i installed and configured the Mosquitto add-on and
started tinkering with my <span class="caps">ESP</span> boards.</p>
</div>
<div class="section" id="logging-the-weather-of-my-office-and-adding-a-doorbell-to-the-mix">
<h2>Logging the “weather” of my office (and adding a doorbell to the mix)</h2>
<p>Since i didn’t want to dismantle the Prometheus based station yet (even though there was no-one to
query it except me calling <tt class="docutils literal">http get <span class="pre">http://192.168.0.12/metrics</span></tt> manually), i decided to
measure the temperature and humidity of my home office environment.</p>
<p>The circuit is <em>really</em> simple:</p>
<ul class="simple">
<li>Take an <span class="caps">ESP8266</span> board and a <span class="caps">DHT22</span> sensor (you can go with a <span class="caps">DHT11</span>, it doesn’t really matter)</li>
<li>Connect the <span class="caps">GND</span>, <span class="caps">VCC</span>, and <span class="caps">OUT</span> pins of the <span class="caps">DHT</span> sensor to the <span class="caps">GND</span>, 3V3, and D5 (any other pin can
work, though)</li>
<li>Install the code below, and behold!</li>
</ul>
<p><span class="caps">OK</span>, i had a bit of precaution, and installed Mosquitto on my local machine, so i could test it
before sending data directly to Hassio.</p>
<p>Meanwhile, i was expecting some package deliveries. I trusted the dogs that they will bark
whenever someone stops at our gate, which i can clearly hear in my office. Well, it turned out
they don’t do that if it’s raining, and since my smartphone rebooted for some unknown reason, the
delivery guy could not reach me and left; he will try to deliver the package again on Monday. All
this happened because i don’t have a doorbell.</p>
<p>So while at it, i quickly installed a push button on our gate, led the wire to my office (a good 5
meters or so), and did some more soldering:</p>
<ul class="simple">
<li>Connect the <span class="caps">GND</span> pin of the <span class="caps">ESP</span> board to D4 through a 10kΩ resistor</li>
<li>Connect one pin of the push button to the 3V3 pin of the <span class="caps">ESP</span> board</li>
<li>Connect the other pin of the push button to the D4 pin of the <span class="caps">ESP</span> board</li>
</ul>
<p>This way D4 is pulled down (ie. remains low) all the time, and is pulled up (becomes high) when
someone pushes the button.</p>
<p>All is good, now to the coding part!</p>
</div>
<div class="section" id="the-code">
<h2>The code</h2>
<p>The code below is already tailored to be used with Home Assistant. While testing, <tt class="docutils literal">BROKER_ADDR</tt>
pointed to my <span class="caps">PC</span>, and <tt class="docutils literal">BROKER_USERNAME</tt> and <tt class="docutils literal">BROKER_PASSWORD</tt> were not defined.</p>
<p>To compile it, you will need the Arduino <span class="caps">IDE</span> (or any toolchain that can compile Arduino code), the
<span class="caps">ESP8266</span> board files for Arduino <span class="caps">IDE</span>, the Adafruit <span class="caps">DHT</span> library, and the <a class="reference external" href="https://github.com/dawidchyrzynski/arduino-home-assistant">home-assistant-integration</a> library.</p>
<div class="highlight"><pre><span></span><span class="cp">#include</span><span class="w"> </span><span class="cpf"><Adafruit_Sensor.h></span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><ESP8266WiFi.h></span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><ArduinoHA.h></span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><DHT.h></span>
<span class="cp">#define LED_PIN BUILTIN_LED</span>
<span class="cp">#define BROKER_ADDR IPAddress(192, 168, 0, 16)</span>
<span class="cp">#define BROKER_USERNAME "homeassistant"</span>
<span class="cp">#define BROKER_PASSWORD "verySecureHomeAssistantMQTTPassword"</span>
<span class="cp">#define BROKER_NAME "office-weather-station"</span>
<span class="cp">#define WIFI_SSID "MyHomeWiFi"</span>
<span class="cp">#define WIFI_PASSWORD "MyHomeWiFiPassword"</span>
<span class="cp">#define DHT_PIN D5</span>
<span class="cp">#define DHT_TYPE DHT22</span>
<span class="cp">#define DOORBELL_PIN D4</span>
<span class="cp">#define UPDATE_INTERVAL 5000</span>
<span class="cp">#define UNIQUE_ID "84cca8aa2673"</span>
<span class="cp">#define BUZZER_PIN D7</span>
<span class="n">WiFiClient</span><span class="w"> </span><span class="n">client</span><span class="p">;</span>
<span class="n">HADevice</span><span class="w"> </span><span class="nf">device</span><span class="p">(</span><span class="n">UNIQUE_ID</span><span class="p">);</span>
<span class="n">HAMqtt</span><span class="w"> </span><span class="nf">mqtt</span><span class="p">(</span><span class="n">client</span><span class="p">,</span><span class="w"> </span><span class="n">device</span><span class="p">);</span>
<span class="n">HASwitch</span><span class="w"> </span><span class="n">led</span><span class="p">(</span><span class="s">"led"</span><span class="p">,</span><span class="w"> </span><span class="nb">false</span><span class="p">,</span><span class="w"> </span><span class="n">mqtt</span><span class="p">);</span>
<span class="n">HABinarySensor</span><span class="w"> </span><span class="n">doorbell</span><span class="p">(</span><span class="s">"doorbell"</span><span class="p">,</span><span class="w"> </span><span class="nb">false</span><span class="p">,</span><span class="w"> </span><span class="n">mqtt</span><span class="p">);</span>
<span class="n">HASensor</span><span class="o"><</span><span class="kt">double</span><span class="o">></span><span class="w"> </span><span class="n">temperature</span><span class="p">(</span><span class="s">"temperature"</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">mqtt</span><span class="p">);</span>
<span class="n">HASensor</span><span class="o"><</span><span class="kt">double</span><span class="o">></span><span class="w"> </span><span class="n">humidity</span><span class="p">(</span><span class="s">"humidity"</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">mqtt</span><span class="p">);</span>
<span class="kt">unsigned</span><span class="w"> </span><span class="kt">long</span><span class="w"> </span><span class="n">last_update</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span>
<span class="n">DHT</span><span class="w"> </span><span class="nf">dht</span><span class="p">(</span><span class="n">DHT_PIN</span><span class="p">,</span><span class="w"> </span><span class="n">DHT_TYPE</span><span class="p">);</span>
<span class="kt">bool</span><span class="w"> </span><span class="n">doorbell_pushed</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">false</span><span class="p">;</span>
<span class="kt">unsigned</span><span class="w"> </span><span class="kt">long</span><span class="w"> </span><span class="n">doorbell_push_time</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span>
<span class="kt">bool</span><span class="w"> </span><span class="n">ringing</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">false</span><span class="p">;</span>
<span class="kt">void</span><span class="w"> </span><span class="nf">onSwitchStateChanged</span><span class="p">(</span><span class="kt">bool</span><span class="w"> </span><span class="n">state</span><span class="p">,</span><span class="w"> </span><span class="n">HASwitch</span><span class="o">*</span><span class="w"> </span><span class="n">s</span><span class="p">)</span>
<span class="p">{</span>
<span class="w"> </span><span class="n">digitalWrite</span><span class="p">(</span><span class="n">LED_PIN</span><span class="p">,</span><span class="w"> </span><span class="p">(</span><span class="n">state</span><span class="w"> </span><span class="o">?</span><span class="w"> </span><span class="n">LOW</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="n">HIGH</span><span class="p">));</span>
<span class="p">}</span>
<span class="kt">void</span><span class="w"> </span><span class="nf">setup</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">sensor_t</span><span class="w"> </span><span class="n">sensor</span><span class="p">;</span>
<span class="w"> </span><span class="n">Serial</span><span class="p">.</span><span class="n">begin</span><span class="p">(</span><span class="mi">9600</span><span class="p">);</span>
<span class="w"> </span><span class="n">Serial</span><span class="p">.</span><span class="n">println</span><span class="p">(</span><span class="s">"Starting..."</span><span class="p">);</span>
<span class="w"> </span><span class="n">dht</span><span class="p">.</span><span class="n">begin</span><span class="p">();</span>
<span class="w"> </span><span class="n">pinMode</span><span class="p">(</span><span class="n">LED_PIN</span><span class="p">,</span><span class="w"> </span><span class="n">OUTPUT</span><span class="p">);</span>
<span class="w"> </span><span class="n">digitalWrite</span><span class="p">(</span><span class="n">LED_PIN</span><span class="p">,</span><span class="w"> </span><span class="n">HIGH</span><span class="p">);</span>
<span class="w"> </span><span class="n">pinMode</span><span class="p">(</span><span class="n">DOORBELL_PIN</span><span class="p">,</span><span class="w"> </span><span class="n">INPUT</span><span class="p">);</span>
<span class="w"> </span><span class="n">pinMode</span><span class="p">(</span><span class="n">BUZZER_PIN</span><span class="p">,</span><span class="w"> </span><span class="n">OUTPUT</span><span class="p">);</span>
<span class="w"> </span><span class="c1">// connect to wifi</span>
<span class="w"> </span><span class="n">WiFi</span><span class="p">.</span><span class="n">begin</span><span class="p">(</span><span class="n">WIFI_SSID</span><span class="p">,</span><span class="w"> </span><span class="n">WIFI_PASSWORD</span><span class="p">);</span>
<span class="w"> </span><span class="k">while</span><span class="w"> </span><span class="p">(</span><span class="n">WiFi</span><span class="p">.</span><span class="n">status</span><span class="p">()</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="n">WL_CONNECTED</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">Serial</span><span class="p">.</span><span class="n">print</span><span class="p">(</span><span class="s">"."</span><span class="p">);</span>
<span class="w"> </span><span class="n">delay</span><span class="p">(</span><span class="mi">500</span><span class="p">);</span><span class="w"> </span><span class="c1">// waiting for the connection</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">Serial</span><span class="p">.</span><span class="n">println</span><span class="p">();</span>
<span class="w"> </span><span class="n">Serial</span><span class="p">.</span><span class="n">print</span><span class="p">(</span><span class="s">"Connected to the network, IP address: "</span><span class="p">);</span>
<span class="w"> </span><span class="n">Serial</span><span class="p">.</span><span class="n">println</span><span class="p">(</span><span class="n">WiFi</span><span class="p">.</span><span class="n">localIP</span><span class="p">());</span>
<span class="w"> </span><span class="c1">// set device's details (optional)</span>
<span class="w"> </span><span class="n">device</span><span class="p">.</span><span class="n">setName</span><span class="p">(</span><span class="n">BROKER_NAME</span><span class="p">);</span>
<span class="w"> </span><span class="n">device</span><span class="p">.</span><span class="n">setSoftwareVersion</span><span class="p">(</span><span class="s">"0.3.0"</span><span class="p">);</span>
<span class="w"> </span><span class="c1">// handle switch state</span>
<span class="w"> </span><span class="n">led</span><span class="p">.</span><span class="n">onStateChanged</span><span class="p">(</span><span class="n">onSwitchStateChanged</span><span class="p">);</span>
<span class="w"> </span><span class="n">temperature</span><span class="p">.</span><span class="n">setUnitOfMeasurement</span><span class="p">(</span><span class="s">"°C"</span><span class="p">);</span>
<span class="w"> </span><span class="n">humidity</span><span class="p">.</span><span class="n">setUnitOfMeasurement</span><span class="p">(</span><span class="s">"%"</span><span class="p">);</span>
<span class="w"> </span><span class="n">mqtt</span><span class="p">.</span><span class="n">begin</span><span class="p">(</span><span class="n">BROKER_ADDR</span>
<span class="cp">#if defined(BROKER_USERNAME) && defined(BROKER_PASSWORD)</span>
<span class="w"> </span><span class="p">,</span><span class="w"> </span><span class="n">BROKER_USERNAME</span><span class="p">,</span><span class="w"> </span><span class="n">BROKER_PASSWORD</span>
<span class="cp">#endif</span>
<span class="w"> </span><span class="p">);</span>
<span class="p">}</span>
<span class="kt">void</span><span class="w"> </span><span class="nf">loop</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">sensors_event_t</span><span class="w"> </span><span class="n">event</span><span class="p">;</span>
<span class="w"> </span><span class="kt">unsigned</span><span class="w"> </span><span class="kt">long</span><span class="w"> </span><span class="n">now</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">millis</span><span class="p">();</span>
<span class="w"> </span><span class="kt">float</span><span class="w"> </span><span class="n">temp_value</span><span class="p">;</span>
<span class="w"> </span><span class="kt">float</span><span class="w"> </span><span class="n">humid_value</span><span class="p">;</span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">doorbell_state</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">digitalRead</span><span class="p">(</span><span class="n">DOORBELL_PIN</span><span class="p">);</span>
<span class="w"> </span><span class="n">mqtt</span><span class="p">.</span><span class="n">loop</span><span class="p">();</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">((</span><span class="n">doorbell_state</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">HIGH</span><span class="p">)</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="o">!</span><span class="n">doorbell_pushed</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">doorbell_pushed</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">true</span><span class="p">;</span>
<span class="w"> </span><span class="n">doorbell_push_time</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">now</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">((</span><span class="n">doorbell_state</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">LOW</span><span class="p">)</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="n">doorbell_pushed</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">doorbell_pushed</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">false</span><span class="p">;</span>
<span class="w"> </span><span class="n">doorbell</span><span class="p">.</span><span class="n">setState</span><span class="p">(</span><span class="nb">false</span><span class="p">);</span>
<span class="w"> </span><span class="n">ringing</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">false</span><span class="p">;</span>
<span class="w"> </span><span class="n">Serial</span><span class="p">.</span><span class="n">println</span><span class="p">(</span><span class="s">"Doorbell released"</span><span class="p">);</span>
<span class="w"> </span><span class="n">analogWrite</span><span class="p">(</span><span class="n">BUZZER_PIN</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">);</span>
<span class="w"> </span><span class="n">delay</span><span class="p">(</span><span class="mi">100</span><span class="p">);</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">doorbell_pushed</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="o">!</span><span class="n">ringing</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="p">(</span><span class="n">now</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">doorbell_push_time</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="mi">100</span><span class="p">))</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">ringing</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">true</span><span class="p">;</span>
<span class="w"> </span><span class="n">Serial</span><span class="p">.</span><span class="n">println</span><span class="p">(</span><span class="s">"Doorbell pushed"</span><span class="p">);</span>
<span class="w"> </span><span class="n">doorbell</span><span class="p">.</span><span class="n">setState</span><span class="p">(</span><span class="nb">true</span><span class="p">);</span>
<span class="w"> </span><span class="n">analogWrite</span><span class="p">(</span><span class="n">BUZZER_PIN</span><span class="p">,</span><span class="w"> </span><span class="mi">255</span><span class="p">);</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">((</span><span class="n">now</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">last_update</span><span class="p">)</span><span class="w"> </span><span class="o">>=</span><span class="w"> </span><span class="n">UPDATE_INTERVAL</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">last_update</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">now</span><span class="p">;</span>
<span class="w"> </span><span class="n">temp_value</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">dht</span><span class="p">.</span><span class="n">readTemperature</span><span class="p">();</span>
<span class="w"> </span><span class="n">temperature</span><span class="p">.</span><span class="n">setValue</span><span class="p">(</span><span class="n">temp_value</span><span class="p">);</span>
<span class="w"> </span><span class="n">Serial</span><span class="p">.</span><span class="n">print</span><span class="p">(</span><span class="s">"Read temperature "</span><span class="p">);</span>
<span class="w"> </span><span class="n">Serial</span><span class="p">.</span><span class="n">println</span><span class="p">(</span><span class="n">temp_value</span><span class="p">);</span>
<span class="w"> </span><span class="n">humid_value</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">dht</span><span class="p">.</span><span class="n">readHumidity</span><span class="p">();</span>
<span class="w"> </span><span class="n">humidity</span><span class="p">.</span><span class="n">setValue</span><span class="p">(</span><span class="n">humid_value</span><span class="p">);</span>
<span class="w"> </span><span class="n">Serial</span><span class="p">.</span><span class="n">print</span><span class="p">(</span><span class="s">"Read humidity "</span><span class="p">);</span>
<span class="w"> </span><span class="n">Serial</span><span class="p">.</span><span class="n">println</span><span class="p">(</span><span class="n">humid_value</span><span class="p">);</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>Nice, isn’t it? Well, can you see that <tt class="docutils literal">UNIQUE_ID</tt> thing? That’s the ugly part, and that’s what
this article is about.</p>
</div>
<div class="section" id="into-the-depths-of-the-home-assistant-integration">
<h2>Into the depths of the <tt class="docutils literal"><span class="pre">home-assistant-integration</span></tt></h2>
<p>The NodeMCU example of the library goes like this (only the relevant parts are here):</p>
<div class="highlight"><pre><span></span><span class="n">byte</span><span class="w"> </span><span class="n">mac</span><span class="p">[</span><span class="mi">6</span><span class="p">];</span>
<span class="n">WiFiClient</span><span class="w"> </span><span class="n">client</span><span class="p">;</span>
<span class="n">HADevice</span><span class="w"> </span><span class="nf">device</span><span class="p">(</span><span class="n">mac</span><span class="p">,</span><span class="w"> </span><span class="k">sizeof</span><span class="p">(</span><span class="n">mac</span><span class="p">));</span>
<span class="n">HAMqtt</span><span class="w"> </span><span class="nf">mqtt</span><span class="p">(</span><span class="n">client</span><span class="p">,</span><span class="w"> </span><span class="n">device</span><span class="p">);</span>
<span class="n">HASwitch</span><span class="w"> </span><span class="n">led</span><span class="p">(</span><span class="s">"led"</span><span class="p">,</span><span class="w"> </span><span class="nb">false</span><span class="p">,</span><span class="w"> </span><span class="n">mqtt</span><span class="p">);</span><span class="w"> </span><span class="c1">// you can use custom name in place of "led"</span>
<span class="kt">void</span><span class="w"> </span><span class="nf">setup</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">WiFi</span><span class="p">.</span><span class="n">macAddress</span><span class="p">(</span><span class="n">mac</span><span class="p">);</span>
<span class="w"> </span><span class="n">WiFi</span><span class="p">.</span><span class="n">begin</span><span class="p">(</span><span class="n">WIFI_SSID</span><span class="p">,</span><span class="w"> </span><span class="n">WIFI_PASSWORD</span><span class="p">);</span>
<span class="w"> </span><span class="n">mqtt</span><span class="p">.</span><span class="n">begin</span><span class="p">(</span><span class="n">BROKER_ADDRESS</span><span class="p">);</span>
<span class="p">}</span>
</pre></div>
<p>Now guess what the unique <span class="caps">ID</span> of the device will be. I’ll wait…</p>
<p>Was your answer “the <span class="caps">MAC</span> address of the <span class="caps">ESP</span> board’s WiFi chip”? Yeah, mine too. Except it will
be <tt class="docutils literal">000000000000</tt>. If you want to install one station in your house, that’s not a big deal.
But i want one outside, one in my office, in the kitchen, the bedroom, bathroom, and so one.
Having the same unique <span class="caps">ID</span> makes it not-so-unique in this case. So I dug deeper in the code of <tt class="docutils literal">HADevice</tt>.</p>
<p>It has the following constructors:</p>
<div class="highlight"><pre><span></span><span class="n">HADevice</span><span class="o">::</span><span class="n">HADevice</span><span class="p">(</span><span class="k">const</span><span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="o">*</span><span class="n">uniqueId</span><span class="p">)</span><span class="w"> </span><span class="o">:</span>
<span class="w"> </span><span class="n">_uniqueId</span><span class="p">(</span><span class="n">uniqueId</span><span class="p">);</span>
<span class="w"> </span><span class="n">HADEVICE_INIT</span>
<span class="p">{}</span>
<span class="kt">uint16_t</span><span class="w"> </span><span class="n">HADevice</span><span class="o">::</span><span class="n">HADevice</span><span class="p">(</span><span class="k">const</span><span class="w"> </span><span class="n">byte</span><span class="w"> </span><span class="o">*</span><span class="n">uniqueId</span><span class="p">,</span><span class="w"> </span><span class="k">const</span><span class="w"> </span><span class="kt">uint16_t</span><span class="w"> </span><span class="o">&</span><span class="n">length</span><span class="p">)</span><span class="w"> </span><span class="o">:</span>
<span class="w"> </span><span class="n">_uniqueId</span><span class="p">(</span><span class="n">HAUtils</span><span class="o">::</span><span class="n">byteArrayToStr</span><span class="p">(</span><span class="n">uniqueId</span><span class="p">,</span><span class="w"> </span><span class="n">length</span><span class="p">)),</span>
<span class="w"> </span><span class="n">HADEVICE_INIT</span>
<span class="p">{}</span>
</pre></div>
<p>Meanwhile, the <tt class="docutils literal">WiFi.macAddress(mac)</tt> line calls a function that <em>gets</em> the <span class="caps">MAC</span> address of the
WiFi chip, and stores the bytes in the <tt class="docutils literal">mac</tt> array.</p>
<p>So what happens? How does the unique <span class="caps">ID</span> become a string of zeroes? Well, the example code calls
the second constructor, effectively converting the <tt class="docutils literal">mac</tt> array (full of zeroes) to a character
string full of zeroes.</p>
</div>
<div class="section" id="the-solution">
<h2>The solution</h2>
<p id="solution">As you can’t get the <span class="caps">MAC</span> address of the WiFi card outside of a function like <tt class="docutils literal">setup()</tt>, we can’t
rely on the second constructor form. However, the first form is more interesting if you look at
the code behind it: if you provide a string as the unique <span class="caps">ID</span>, it will be used without any
mangling. So let’s update our code a bit:</p>
<div class="highlight"><pre><span></span><span class="cp">#include</span><span class="w"> </span><span class="cpf"><ESP8266WiFi.h></span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><ArduinoHA.h></span>
<span class="cp">#define LED_PIN D0</span>
<span class="cp">#define BROKER_ADDR IPAddress(192, 168, 0, 17)</span>
<span class="cp">#define WIFI_SSID "MyNetwork"</span>
<span class="cp">#define WIFI_PASSWORD "MyPassword"</span>
<span class="kt">char</span><span class="w"> </span><span class="n">macaddress</span><span class="p">[</span><span class="mi">13</span><span class="p">];</span>
<span class="n">WiFiClient</span><span class="w"> </span><span class="n">client</span><span class="p">;</span>
<span class="n">HADevice</span><span class="w"> </span><span class="nf">device</span><span class="p">(</span><span class="n">macaddress</span><span class="p">);</span>
<span class="n">HAMqtt</span><span class="w"> </span><span class="nf">mqtt</span><span class="p">(</span><span class="n">client</span><span class="p">,</span><span class="w"> </span><span class="n">device</span><span class="p">);</span>
<span class="n">HASwitch</span><span class="w"> </span><span class="n">led</span><span class="p">(</span><span class="s">"led"</span><span class="p">,</span><span class="w"> </span><span class="nb">false</span><span class="p">,</span><span class="w"> </span><span class="n">mqtt</span><span class="p">);</span><span class="w"> </span><span class="c1">// you can use custom name in place of "led"</span>
<span class="kt">void</span><span class="w"> </span><span class="nf">onSwitchStateChanged</span><span class="p">(</span><span class="kt">bool</span><span class="w"> </span><span class="n">state</span><span class="p">,</span><span class="w"> </span><span class="n">HASwitch</span><span class="o">*</span><span class="w"> </span><span class="n">s</span><span class="p">)</span>
<span class="p">{</span>
<span class="w"> </span><span class="n">digitalWrite</span><span class="p">(</span><span class="n">LED_PIN</span><span class="p">,</span><span class="w"> </span><span class="p">(</span><span class="n">state</span><span class="w"> </span><span class="o">?</span><span class="w"> </span><span class="n">HIGH</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="n">LOW</span><span class="p">));</span>
<span class="p">}</span>
<span class="kt">void</span><span class="w"> </span><span class="nf">setup</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">byte</span><span class="w"> </span><span class="n">mac</span><span class="p">[</span><span class="mi">6</span><span class="p">];</span>
<span class="w"> </span><span class="n">Serial</span><span class="p">.</span><span class="n">begin</span><span class="p">(</span><span class="mi">9600</span><span class="p">);</span>
<span class="w"> </span><span class="n">Serial</span><span class="p">.</span><span class="n">println</span><span class="p">(</span><span class="s">"Starting..."</span><span class="p">);</span>
<span class="w"> </span><span class="n">WiFi</span><span class="p">.</span><span class="n">macAddress</span><span class="p">(</span><span class="n">mac</span><span class="p">);</span>
<span class="w"> </span><span class="n">pinMode</span><span class="p">(</span><span class="n">LED_PIN</span><span class="p">,</span><span class="w"> </span><span class="n">OUTPUT</span><span class="p">);</span>
<span class="w"> </span><span class="n">digitalWrite</span><span class="p">(</span><span class="n">LED_PIN</span><span class="p">,</span><span class="w"> </span><span class="n">LOW</span><span class="p">);</span>
<span class="w"> </span><span class="c1">// connect to wifi</span>
<span class="w"> </span><span class="n">WiFi</span><span class="p">.</span><span class="n">begin</span><span class="p">(</span><span class="n">WIFI_SSID</span><span class="p">,</span><span class="w"> </span><span class="n">WIFI_PASSWORD</span><span class="p">);</span>
<span class="w"> </span><span class="k">while</span><span class="w"> </span><span class="p">(</span><span class="n">WiFi</span><span class="p">.</span><span class="n">status</span><span class="p">()</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="n">WL_CONNECTED</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">Serial</span><span class="p">.</span><span class="n">print</span><span class="p">(</span><span class="s">"."</span><span class="p">);</span>
<span class="w"> </span><span class="n">delay</span><span class="p">(</span><span class="mi">500</span><span class="p">);</span><span class="w"> </span><span class="c1">// waiting for the connection</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">Serial</span><span class="p">.</span><span class="n">println</span><span class="p">();</span>
<span class="w"> </span><span class="n">Serial</span><span class="p">.</span><span class="n">println</span><span class="p">(</span><span class="s">"Connected to the network"</span><span class="p">);</span>
<span class="w"> </span><span class="n">snprintf</span><span class="p">(</span><span class="n">macaddress</span><span class="p">,</span><span class="w"> </span><span class="mi">13</span><span class="p">,</span>
<span class="w"> </span><span class="s">"%02x%02x%02x%02x%02x%02x"</span><span class="p">,</span>
<span class="w"> </span><span class="n">mac</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span><span class="w"> </span><span class="n">mac</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span><span class="w"> </span><span class="n">mac</span><span class="p">[</span><span class="mi">2</span><span class="p">],</span><span class="w"> </span><span class="n">mac</span><span class="p">[</span><span class="mi">3</span><span class="p">],</span><span class="w"> </span><span class="n">mac</span><span class="p">[</span><span class="mi">4</span><span class="p">],</span><span class="w"> </span><span class="n">mac</span><span class="p">[</span><span class="mi">5</span><span class="p">]);</span>
<span class="w"> </span><span class="c1">// set device's details (optional)</span>
<span class="w"> </span><span class="n">device</span><span class="p">.</span><span class="n">setName</span><span class="p">(</span><span class="s">"NodeMCU"</span><span class="p">);</span>
<span class="w"> </span><span class="n">device</span><span class="p">.</span><span class="n">setSoftwareVersion</span><span class="p">(</span><span class="s">"1.0.0"</span><span class="p">);</span>
<span class="w"> </span><span class="c1">// handle switch state</span>
<span class="w"> </span><span class="n">led</span><span class="p">.</span><span class="n">onStateChanged</span><span class="p">(</span><span class="n">onSwitchStateChanged</span><span class="p">);</span>
<span class="w"> </span><span class="n">mqtt</span><span class="p">.</span><span class="n">begin</span><span class="p">(</span><span class="n">BROKER_ADDR</span><span class="p">);</span>
<span class="p">}</span>
<span class="kt">void</span><span class="w"> </span><span class="nf">loop</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">mqtt</span><span class="p">.</span><span class="n">loop</span><span class="p">();</span>
<span class="p">}</span>
</pre></div>
<p>And now you have a unique(ish) <span class="caps">ID</span> (well, unless you start tinkering with <span class="caps">MAC</span> addresses on you
network, but then you are on your own). And if you want to update the unique <span class="caps">ID</span> while the
software is still running, you can do that, too. But i won’t help you with such perversions.</p>
</div>
Converting werkzeug’s hashes to Passlib format2020-07-19T05:02:00+00:002020-07-19T05:02:00+00:00Gergely Polonkaitag:None,2020-07-19:blog/converting-werkzeug-hashes-to-passlib/<p>In one of my projects i use <a class="reference external" href="https://palletsprojects.com/p/werkzeug/">Werkzeug</a>’s
<tt class="docutils literal">generate_password_hash</tt> to, well, generate password hashes. Werkzeug uses Python’s
<a class="reference external" href="https://docs.python.org/3.8/library/hashlib.html">hashlib</a> module under the hood which,
unfortunately to me, doesn’t support Argon2 (which i want to transition to for greater security).</p>
<p>Enter <a class="reference external" href="https://passlib.readthedocs.io/">Passlib</a>.</p>
<p>Passlib conveniently supports both <a class="reference external" href="https://passlib.readthedocs.io/en/stable/lib/passlib.hash.pbkdf2_digest.html"><span class="caps">PBKDF2</span></a> (what i …</p><p>In one of my projects i use <a class="reference external" href="https://palletsprojects.com/p/werkzeug/">Werkzeug</a>’s
<tt class="docutils literal">generate_password_hash</tt> to, well, generate password hashes. Werkzeug uses Python’s
<a class="reference external" href="https://docs.python.org/3.8/library/hashlib.html">hashlib</a> module under the hood which,
unfortunately to me, doesn’t support Argon2 (which i want to transition to for greater security).</p>
<p>Enter <a class="reference external" href="https://passlib.readthedocs.io/">Passlib</a>.</p>
<p>Passlib conveniently supports both <a class="reference external" href="https://passlib.readthedocs.io/en/stable/lib/passlib.hash.pbkdf2_digest.html"><span class="caps">PBKDF2</span></a> (what i currently
use in the project, using <span class="caps">SHA256</span> digests), and also supports <a class="reference external" href="https://passlib.readthedocs.io/en/stable/lib/passlib.hash.argon2.html">Argon2</a>. Here are the differences:</p>
<table border="1" class="docutils">
<thead valign="bottom">
<tr><th class="head">Feature</th>
<th class="head">Werkzeug</th>
<th class="head">Passlib</th>
</tr>
</thead>
<tbody valign="top">
<tr><td>Iteration count</td>
<td>150000, hardcoded</td>
<td>29000, changeable</td>
</tr>
<tr><td>Default method</td>
<td><span class="caps">PBKDF2</span></td>
<td>None, must be set explicitly</td>
</tr>
<tr><td>Default digest</td>
<td><span class="caps">SHA</span>-256</td>
<td>None, must be set explicitly</td>
</tr>
<tr><td>Default salt size</td>
<td>8, changeable</td>
<td>16, changeable</td>
</tr>
<tr><td>Salt character set</td>
<td><span class="caps">ASCII</span> only</td>
<td>Binary</td>
</tr>
<tr><td>Salt storage</td>
<td>Plain text (shorter)</td>
<td>Adapted Base64</td>
</tr>
<tr><td>Hash storage</td>
<td>Hex string</td>
<td>Adapted Base64 (shorter)</td>
</tr>
</tbody>
</table>
<p>But even if i force the same settings on Passlib, their format is different and, obviously,
Passlib doesn’t understand Werkzeug’s format so i had to convert all my hashes to the new format.</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="n">generate_password_hash</span><span class="p">(</span><span class="s1">'password'</span><span class="p">,</span> <span class="n">method</span><span class="o">=</span><span class="s1">'pbkdf2:sha256'</span><span class="p">,</span> <span class="n">salt_length</span><span class="o">=</span><span class="mi">8</span><span class="p">)</span>
<span class="s1">'pbkdf2:sha256:150000$2dFFA24B$1602ed71733451acaf29abd26b1d1a78aced4442467f9efbab9b1fa21ae39d68'</span>
<span class="o">>>></span> <span class="n">pbkdf2_sha256</span><span class="o">.</span><span class="n">using</span><span class="p">(</span><span class="n">rounds</span><span class="o">=</span><span class="mi">150000</span><span class="p">,</span> <span class="n">salt_size</span><span class="o">=</span><span class="mi">8</span><span class="p">)</span><span class="o">.</span><span class="n">hash</span><span class="p">(</span><span class="s1">'password'</span><span class="p">)</span>
<span class="s1">'$pbkdf2-sha256$29000$tdZ6731vzTnn/H9vTSnl3A$maWqohBS0ghQEjIoJWnYC1zGF2T/qOwRmHzVzHt3NqU'</span>
</pre></div>
<p>First, let’s split the old hash by the <tt class="docutils literal">$</tt> characters, and also split the first part by colons:</p>
<div class="highlight"><pre><span></span><span class="n">full_method</span><span class="p">,</span> <span class="n">salt</span><span class="p">,</span> <span class="n">hashed_value</span> <span class="o">=</span> <span class="n">old_hash</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">'$'</span><span class="p">)</span>
<span class="n">method</span><span class="p">,</span> <span class="n">digest</span><span class="p">,</span> <span class="n">rounds</span> <span class="o">=</span> <span class="n">full_method</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">':'</span><span class="p">)</span>
</pre></div>
<p>As it soon turned out, the two libraries even store the actual data in different formats.
Werkzeug stores the salt in plain text (it’s always <span class="caps">ASCII</span> characters), and the resulting hash in a
hex string. Passlib, however, aims for greater security with a binary salt, so it’s Base64
encoded. This encoding is, however, a bit different from what you’d expect, as it’s “using
shortened base64 format which omits padding <span class="amp">&</span> whitespace” and also “uses custom <tt class="docutils literal">./</tt> altchars”.
It even has its own <a class="reference external" href="https://passlib.readthedocs.io/en/stable/lib/passlib.utils.binary.html?highlight=ab64_encode#passlib.utils.binary.ab64_encode">ab64_encode</a>
and <a class="reference external" href="https://passlib.readthedocs.io/en/stable/lib/passlib.utils.binary.html?highlight=ab64_encode#passlib.utils.binary.ab64_decode">ab64_decode</a>
functions to handle this encoding. So i first need to re-encode both the salt string and the hash
value hex string, so i have raw bytes.</p>
<div class="highlight"><pre><span></span><span class="n">salt_bytes</span> <span class="o">=</span> <span class="n">salt</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">'ascii'</span><span class="p">)</span>
<span class="n">hash_bytes</span> <span class="o">=</span> <span class="nb">bytes</span><span class="o">.</span><span class="n">fromhex</span><span class="p">(</span><span class="nb">hash</span><span class="p">)</span>
</pre></div>
<p>Then encode them to the adapted Base64 format (i also convert it back to <tt class="docutils literal">str</tt> for easier
concatenation later):</p>
<div class="highlight"><pre><span></span><span class="n">passlib_salt</span> <span class="o">=</span> <span class="n">ab64_encode</span><span class="p">(</span><span class="n">salt_bytes</span><span class="p">)</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">'ascii'</span><span class="p">)</span>
<span class="n">passlib_hash</span> <span class="o">=</span> <span class="n">ab64_encode</span><span class="p">(</span><span class="n">hash_bytes</span><span class="p">)</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">'ascii'</span><span class="p">)</span>
</pre></div>
<p>Now we just need to concatenate all the things with the right separators.</p>
<div class="highlight"><pre><span></span><span class="n">passlib_final_hash</span> <span class="o">=</span> <span class="sa">f</span><span class="s1">'$</span><span class="si">{</span><span class="n">method</span><span class="si">}</span><span class="s1">-</span><span class="si">{</span><span class="n">digest</span><span class="si">}</span><span class="s1">$</span><span class="si">{</span><span class="n">rounds</span><span class="si">}</span><span class="s1">$</span><span class="si">{</span><span class="n">passlib_salt</span><span class="si">}</span><span class="s1">$</span><span class="si">{</span><span class="n">passlib_hash</span><span class="si">}</span><span class="s1">'</span>
</pre></div>
<p>Finally, let’s verify everything went right:</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="n">pbkdf2_sha256</span><span class="o">.</span><span class="n">verify</span><span class="p">(</span><span class="s1">'password'</span><span class="p">,</span> <span class="n">passlib_final_hash</span><span class="p">)</span>
<span class="kc">True</span>
</pre></div>
<p>Here’s the whole series of command, converted to a Python function (with slightly altered variable
names) for your copy-pasting convenience (plus, it’s not using f-strings, so you can use it with
Python <3.6):</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">werkzeug_to_passlib_hash</span><span class="p">(</span><span class="n">old_hash</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Convert Werkzeug’s password hashes to Passlib format.</span>
<span class="sd"> Copied from https://gergely.polonkai.eu/blog/converting-werkzeug-hashes-to-passlib/</span>
<span class="sd"> """</span>
<span class="kn">from</span> <span class="nn">passlib.utils</span> <span class="kn">import</span> <span class="n">ab64_encode</span>
<span class="c1"># Werkzeug hashes look like full_method$salt$hash. We handle full_method later; salt is</span>
<span class="c1"># an ASCII string; hashed value is the hex string representation of the hashed value</span>
<span class="n">full_method</span><span class="p">,</span> <span class="n">salt</span><span class="p">,</span> <span class="n">hashed_value</span> <span class="o">=</span> <span class="n">old_hash</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">'$'</span><span class="p">)</span>
<span class="c1"># Werkzeug’s full_method is a colon delimited list of the method, digest, and rounds</span>
<span class="n">method</span><span class="p">,</span> <span class="n">digest</span><span class="p">,</span> <span class="n">rounds</span> <span class="o">=</span> <span class="n">full_method</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">':'</span><span class="p">)</span>
<span class="n">new_parts</span> <span class="o">=</span> <span class="p">[</span>
<span class="c1"># Passlib expects the hashed value to starts with a $ sign (hence the empty string at the</span>
<span class="c1"># beginning of this list).</span>
<span class="s1">''</span><span class="p">,</span>
<span class="c1"># Passlib’s method and digest is concatenated together with a hyphen.</span>
<span class="sa">f</span><span class="s1">'</span><span class="si">{</span><span class="n">method</span><span class="si">}</span><span class="s1">-</span><span class="si">{</span><span class="n">digest</span><span class="si">}</span><span class="s1">'</span><span class="p">,</span>
<span class="n">rounds</span><span class="p">,</span>
<span class="c1"># Passlib expects the salt and the actual hash to be in the adapted base64 encoding format.</span>
<span class="n">ab64_encode</span><span class="p">(</span><span class="n">salt</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">'ascii'</span><span class="p">))</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">'ascii'</span><span class="p">),</span>
<span class="n">ab64_encode</span><span class="p">(</span><span class="nb">bytes</span><span class="o">.</span><span class="n">fromhex</span><span class="p">(</span><span class="n">hashed_value</span><span class="p">))</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">'utf-8'</span><span class="p">),</span>
<span class="p">]</span>
<span class="k">return</span> <span class="s1">'$'</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">new_parts</span><span class="p">)</span>
</pre></div>
Mastodon database hickup2020-04-23T11:17:00+00:002020-04-23T11:17:00+00:00Gergely Polonkaitag:None,2020-04-23:blog/mastodon-database-hickup/<p>After being online for almost a year, my <a class="reference external" href="https://social.polonkai.eu">Mastodon instance</a> is back
up. However, it wasn’t as straightforward as i wanted it to be.</p>
<p>About a year ago my Mastodon server went down. I messed up my docker images big time and the
then-new version 2.8.4 didn …</p><p>After being online for almost a year, my <a class="reference external" href="https://social.polonkai.eu">Mastodon instance</a> is back
up. However, it wasn’t as straightforward as i wanted it to be.</p>
<p>About a year ago my Mastodon server went down. I messed up my docker images big time and the
then-new version 2.8.4 didn’t want to start up. I left it there, and as i had a pretty rough year
(in the good sense) i didn’t have time to bring it up again.</p>
<p>One of the constant problems i had with this instance is the size of the attachment directory. It
easily grew to an enormous size in no time, so i had to constantly clean things up. Plus, it took
away the precious space away from my other projects. So my first move was moving all these files
to a DigitalOcean Space; with the <tt class="docutils literal">awscli</tt> Python package it was easy as pie. If you want to do
it, too, i followed <a class="reference external" href="https://angristan.xyz/2018/05/moving-mastodon-media-files-to-wasabi-object-storage/">this article</a>.</p>
<p>Next came the database.</p>
<p>I tried to load the database dump, created using <tt class="docutils literal">pg_dump <span class="pre">-f</span> mastodon.sql mastodon</tt>, with
<tt class="docutils literal">psql <span class="pre">-f</span> mastodon.sql</tt>. Everything went fine, all the records created, except this error message:</p>
<div class="highlight"><pre><span></span>psql:mastodon-dump.sql:1691661: ERROR: could not create unique index "index_tags_on_name"
DETAIL: Key (name)=(opensource) is duplicated.
</pre></div>
<p>First i tried to ignore it, but running <cite>rails db:migrate</cite> failed when it wanted to manipulate
this index.</p>
<p>So i issued this query:</p>
<div class="highlight"><pre><span></span><span class="k">SELECT</span><span class="w"> </span><span class="k">COUNT</span><span class="p">(</span><span class="o">*</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="k">count</span><span class="p">,</span><span class="w"> </span><span class="n">name</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">tags</span><span class="w"> </span><span class="k">GROUP</span><span class="w"> </span><span class="k">BY</span><span class="w"> </span><span class="n">name</span><span class="w"> </span><span class="k">HAVING</span><span class="w"> </span><span class="k">COUNT</span><span class="p">(</span><span class="o">*</span><span class="p">)</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="mi">1</span><span class="p">;</span>
</pre></div>
<p>And the result was this:</p>
<div class="highlight"><pre><span></span>count | name
2 | opensource
2 | xmpp
2 | fdroid
2 | socialmedia
2 | blogging
2 | c64
2 | blog
</pre></div>
<p>Then collected the IDs of these duplicate tag records using this query:</p>
<div class="highlight"><pre><span></span><span class="k">SELECT</span><span class="w"> </span><span class="n">id</span><span class="p">,</span><span class="w"> </span><span class="n">name</span><span class="p">,</span><span class="w"> </span><span class="n">created_at</span>
<span class="k">FROM</span><span class="w"> </span><span class="n">tags</span>
<span class="k">WHERE</span><span class="w"> </span><span class="n">name</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="p">(</span><span class="s1">'opensource'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'xmpp'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'fdroid'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'socialmedia'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'blogging'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'c64'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'blog'</span><span class="p">)</span>
<span class="k">ORDER</span><span class="w"> </span><span class="k">BY</span><span class="w"> </span><span class="n">name</span><span class="p">,</span><span class="w"> </span><span class="n">created_at</span><span class="p">;</span>
</pre></div>
<div class="highlight"><pre><span></span> id | name | created_at
-------+-------------+------------
159 | blog | 2018-02-24 19:56:01.710702
15941 | blog | 2019-04-03 22:05:56.438488
158 | blogging | 2018-02-24 19:56:01.721354
16006 | blogging | 2019-04-09 12:13:20.852976
5441 | c64 | 2018-07-31 00:50:56.172468
16036 | c64 | 2019-04-14 19:56:40.692197
924 | fdroid | 2018-04-14 19:39:21.261817
15947 | fdroid | 2019-04-04 19:10:50.190317
237 | opensource | 2018-03-05 21:50:52.609723
15929 | opensource | 2019-04-03 11:49:21.772961
251 | socialmedia | 2018-03-06 23:02:33.573775
16034 | socialmedia | 2019-04-14 19:07:37.081635
519 | xmpp | 2018-03-28 16:02:05.023784
15988 | xmpp | 2019-04-07 08:06:59.429965
</pre></div>
<p>I wanted to see the <tt class="docutils literal">created_at</tt> field just to know when things got bad. As it turns out, it
was around April 2019, so it might have happened with the 2.8.0 upgrade. Was it me, or some
problem with the migrations remains a mistery.</p>
<p>I also wanted to know about all the tables referencing tags:</p>
<div class="highlight"><pre><span></span><span class="k">SELECT</span>
<span class="w"> </span><span class="n">tc</span><span class="p">.</span><span class="k">table_name</span><span class="p">,</span><span class="w"> </span><span class="n">kcu</span><span class="p">.</span><span class="k">column_name</span>
<span class="k">FROM</span>
<span class="w"> </span><span class="n">information_schema</span><span class="p">.</span><span class="n">table_constraints</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">tc</span>
<span class="w"> </span><span class="k">JOIN</span><span class="w"> </span><span class="n">information_schema</span><span class="p">.</span><span class="n">key_column_usage</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">kcu</span>
<span class="w"> </span><span class="k">ON</span><span class="w"> </span><span class="n">tc</span><span class="p">.</span><span class="k">constraint_name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">kcu</span><span class="p">.</span><span class="k">constraint_name</span>
<span class="w"> </span><span class="k">JOIN</span><span class="w"> </span><span class="n">information_schema</span><span class="p">.</span><span class="n">constraint_column_usage</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">ccu</span>
<span class="w"> </span><span class="k">ON</span><span class="w"> </span><span class="n">ccu</span><span class="p">.</span><span class="k">constraint_name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">tc</span><span class="p">.</span><span class="k">constraint_name</span>
<span class="k">WHERE</span><span class="w"> </span><span class="n">constraint_type</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'FOREIGN KEY'</span><span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="n">ccu</span><span class="p">.</span><span class="k">table_name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'tags'</span><span class="p">;</span>
</pre></div>
<div class="highlight"><pre><span></span> table_name | column_name
-------------------+-------------
statuses_tags | tag_id
account_tag_stats | tag_id
</pre></div>
<p>And last, but not least, i had to massage the dump a bit. After creating a backup, i opened up
the file in ViM and looked for the place where it was loading the data into the <tt class="docutils literal">tags</tt> table.
It looked like this:</p>
<div class="highlight"><pre><span></span><span class="k">COPY</span><span class="w"> </span><span class="k">public</span><span class="p">.</span><span class="n">tags</span><span class="w"> </span><span class="p">(</span><span class="n">name</span><span class="p">,</span><span class="w"> </span><span class="n">created_at</span><span class="p">,</span><span class="w"> </span><span class="n">updated_at</span><span class="p">,</span><span class="w"> </span><span class="n">id</span><span class="p">)</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="k">stdin</span><span class="p">;</span>
<span class="n">blogging</span><span class="w"> </span><span class="mi">2018</span><span class="o">-</span><span class="mi">02</span><span class="o">-</span><span class="mi">24</span><span class="w"> </span><span class="mi">19</span><span class="p">:</span><span class="mi">56</span><span class="p">:</span><span class="mi">01</span><span class="p">.</span><span class="mi">710702</span><span class="w"> </span><span class="mi">2018</span><span class="o">-</span><span class="mi">02</span><span class="o">-</span><span class="mi">24</span><span class="w"> </span><span class="mi">19</span><span class="p">:</span><span class="mi">56</span><span class="p">:</span><span class="mi">01</span><span class="p">.</span><span class="mi">710702</span><span class="w"> </span><span class="mi">158</span>
<span class="n">blog</span><span class="w"> </span><span class="mi">2018</span><span class="o">-</span><span class="mi">02</span><span class="o">-</span><span class="mi">24</span><span class="w"> </span><span class="mi">19</span><span class="p">:</span><span class="mi">56</span><span class="p">:</span><span class="mi">01</span><span class="p">.</span><span class="mi">721354</span><span class="w"> </span><span class="mi">2018</span><span class="o">-</span><span class="mi">02</span><span class="o">-</span><span class="mi">24</span><span class="w"> </span><span class="mi">19</span><span class="p">:</span><span class="mi">56</span><span class="p">:</span><span class="mi">01</span><span class="p">.</span><span class="mi">721354</span><span class="w"> </span><span class="mi">159</span>
<span class="n">opensource</span><span class="w"> </span><span class="mi">2018</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">05</span><span class="w"> </span><span class="mi">21</span><span class="p">:</span><span class="mi">50</span><span class="p">:</span><span class="mi">52</span><span class="p">.</span><span class="mi">609723</span><span class="w"> </span><span class="mi">2018</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">05</span><span class="w"> </span><span class="mi">21</span><span class="p">:</span><span class="mi">50</span><span class="p">:</span><span class="mi">52</span><span class="p">.</span><span class="mi">609723</span><span class="w"> </span><span class="mi">237</span>
<span class="n">socialmedia</span><span class="w"> </span><span class="mi">2018</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">06</span><span class="w"> </span><span class="mi">23</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">33</span><span class="p">.</span><span class="mi">573775</span><span class="w"> </span><span class="mi">2018</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">06</span><span class="w"> </span><span class="mi">23</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">33</span><span class="p">.</span><span class="mi">573775</span><span class="w"> </span><span class="mi">251</span>
<span class="n">xmpp</span><span class="w"> </span><span class="mi">2018</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">28</span><span class="w"> </span><span class="mi">16</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">05</span><span class="p">.</span><span class="mi">023784</span><span class="w"> </span><span class="mi">2018</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">28</span><span class="w"> </span><span class="mi">16</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">05</span><span class="p">.</span><span class="mi">023784</span><span class="w"> </span><span class="mi">519</span>
<span class="n">fdroid</span><span class="w"> </span><span class="mi">2018</span><span class="o">-</span><span class="mi">04</span><span class="o">-</span><span class="mi">14</span><span class="w"> </span><span class="mi">19</span><span class="p">:</span><span class="mi">39</span><span class="p">:</span><span class="mi">21</span><span class="p">.</span><span class="mi">261817</span><span class="w"> </span><span class="mi">2018</span><span class="o">-</span><span class="mi">04</span><span class="o">-</span><span class="mi">14</span><span class="w"> </span><span class="mi">19</span><span class="p">:</span><span class="mi">39</span><span class="p">:</span><span class="mi">21</span><span class="p">.</span><span class="mi">261817</span><span class="w"> </span><span class="mi">924</span>
<span class="n">c64</span><span class="w"> </span><span class="mi">2018</span><span class="o">-</span><span class="mi">07</span><span class="o">-</span><span class="mi">31</span><span class="w"> </span><span class="mi">00</span><span class="p">:</span><span class="mi">50</span><span class="p">:</span><span class="mi">56</span><span class="p">.</span><span class="mi">172468</span><span class="w"> </span><span class="mi">2018</span><span class="o">-</span><span class="mi">07</span><span class="o">-</span><span class="mi">31</span><span class="w"> </span><span class="mi">00</span><span class="p">:</span><span class="mi">50</span><span class="p">:</span><span class="mi">56</span><span class="p">.</span><span class="mi">172468</span><span class="w"> </span><span class="mi">5441</span>
<span class="n">opensource</span><span class="w"> </span><span class="mi">2019</span><span class="o">-</span><span class="mi">04</span><span class="o">-</span><span class="mi">03</span><span class="w"> </span><span class="mi">11</span><span class="p">:</span><span class="mi">49</span><span class="p">:</span><span class="mi">21</span><span class="p">.</span><span class="mi">772961</span><span class="w"> </span><span class="mi">2019</span><span class="o">-</span><span class="mi">04</span><span class="o">-</span><span class="mi">03</span><span class="w"> </span><span class="mi">11</span><span class="p">:</span><span class="mi">49</span><span class="p">:</span><span class="mi">21</span><span class="p">.</span><span class="mi">772961</span><span class="w"> </span><span class="mi">15929</span>
<span class="n">blog</span><span class="w"> </span><span class="mi">2019</span><span class="o">-</span><span class="mi">04</span><span class="o">-</span><span class="mi">03</span><span class="w"> </span><span class="mi">22</span><span class="p">:</span><span class="mi">05</span><span class="p">:</span><span class="mi">56</span><span class="p">.</span><span class="mi">438488</span><span class="w"> </span><span class="mi">2019</span><span class="o">-</span><span class="mi">04</span><span class="o">-</span><span class="mi">03</span><span class="w"> </span><span class="mi">22</span><span class="p">:</span><span class="mi">05</span><span class="p">:</span><span class="mi">56</span><span class="p">.</span><span class="mi">438488</span><span class="w"> </span><span class="mi">15941</span>
<span class="n">fdroid</span><span class="w"> </span><span class="mi">2019</span><span class="o">-</span><span class="mi">04</span><span class="o">-</span><span class="mi">04</span><span class="w"> </span><span class="mi">19</span><span class="p">:</span><span class="mi">10</span><span class="p">:</span><span class="mi">50</span><span class="p">.</span><span class="mi">190317</span><span class="w"> </span><span class="mi">2019</span><span class="o">-</span><span class="mi">04</span><span class="o">-</span><span class="mi">04</span><span class="w"> </span><span class="mi">19</span><span class="p">:</span><span class="mi">10</span><span class="p">:</span><span class="mi">50</span><span class="p">.</span><span class="mi">190317</span><span class="w"> </span><span class="mi">15947</span>
<span class="n">xmpp</span><span class="w"> </span><span class="mi">2019</span><span class="o">-</span><span class="mi">04</span><span class="o">-</span><span class="mi">07</span><span class="w"> </span><span class="mi">08</span><span class="p">:</span><span class="mi">06</span><span class="p">:</span><span class="mi">59</span><span class="p">.</span><span class="mi">429965</span><span class="w"> </span><span class="mi">2019</span><span class="o">-</span><span class="mi">04</span><span class="o">-</span><span class="mi">07</span><span class="w"> </span><span class="mi">08</span><span class="p">:</span><span class="mi">06</span><span class="p">:</span><span class="mi">59</span><span class="p">.</span><span class="mi">429965</span><span class="w"> </span><span class="mi">15988</span>
<span class="n">blogging</span><span class="w"> </span><span class="mi">2019</span><span class="o">-</span><span class="mi">04</span><span class="o">-</span><span class="mi">09</span><span class="w"> </span><span class="mi">12</span><span class="p">:</span><span class="mi">13</span><span class="p">:</span><span class="mi">20</span><span class="p">.</span><span class="mi">852976</span><span class="w"> </span><span class="mi">2019</span><span class="o">-</span><span class="mi">04</span><span class="o">-</span><span class="mi">09</span><span class="w"> </span><span class="mi">12</span><span class="p">:</span><span class="mi">13</span><span class="p">:</span><span class="mi">20</span><span class="p">.</span><span class="mi">852976</span><span class="w"> </span><span class="mi">16006</span>
<span class="n">socialmedia</span><span class="w"> </span><span class="mi">2019</span><span class="o">-</span><span class="mi">04</span><span class="o">-</span><span class="mi">14</span><span class="w"> </span><span class="mi">19</span><span class="p">:</span><span class="mi">07</span><span class="p">:</span><span class="mi">37</span><span class="p">.</span><span class="mi">081635</span><span class="w"> </span><span class="mi">2019</span><span class="o">-</span><span class="mi">04</span><span class="o">-</span><span class="mi">14</span><span class="w"> </span><span class="mi">19</span><span class="p">:</span><span class="mi">07</span><span class="p">:</span><span class="mi">37</span><span class="p">.</span><span class="mi">081635</span><span class="w"> </span><span class="mi">16034</span>
<span class="n">c64</span><span class="w"> </span><span class="mi">2019</span><span class="o">-</span><span class="mi">04</span><span class="o">-</span><span class="mi">14</span><span class="w"> </span><span class="mi">19</span><span class="p">:</span><span class="mi">56</span><span class="p">:</span><span class="mi">40</span><span class="p">.</span><span class="mi">692197</span><span class="w"> </span><span class="mi">2019</span><span class="o">-</span><span class="mi">04</span><span class="o">-</span><span class="mi">14</span><span class="w"> </span><span class="mi">19</span><span class="p">:</span><span class="mi">56</span><span class="p">:</span><span class="mi">40</span><span class="p">.</span><span class="mi">692197</span><span class="w"> </span><span class="mi">16036</span>
</pre></div>
<p>For every tag, i removed the line with the latest date, and noted their IDs, then went on to the
<tt class="docutils literal">statuses_tags</tt> table, which looked like this (i don’t put all rows here as it would take a lot
of space):</p>
<div class="highlight"><pre><span></span><span class="k">COPY</span><span class="w"> </span><span class="k">public</span><span class="p">.</span><span class="n">statuses_tags</span><span class="w"> </span><span class="p">(</span><span class="n">status_id</span><span class="p">,</span><span class="w"> </span><span class="n">tag_id</span><span class="p">)</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="k">stdin</span><span class="p">;</span>
<span class="mi">101862091116861098</span><span class="w"> </span><span class="mi">15929</span>
</pre></div>
<p>Here, i had to look for the IDs noted before in the last column (ie. at the end of a line), and
for every line i change the newer <span class="caps">ID</span> to the old ones (15929 became 237, and so on).</p>
<p>Finally, the <tt class="docutils literal">account_tag_stats</tt> table:</p>
<div class="highlight"><pre><span></span><span class="k">COPY</span><span class="w"> </span><span class="k">public</span><span class="p">.</span><span class="n">account_tag_stats</span><span class="w"> </span><span class="p">(</span><span class="n">id</span><span class="p">,</span><span class="w"> </span><span class="n">tag_id</span><span class="p">,</span><span class="w"> </span><span class="n">accounts_count</span><span class="p">,</span><span class="w"> </span><span class="n">hidden</span><span class="p">,</span><span class="w"> </span><span class="n">created_at</span><span class="p">,</span><span class="w"> </span><span class="n">updated_at</span><span class="p">)</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="k">stdin</span><span class="p">;</span>
<span class="mi">1</span><span class="w"> </span><span class="mi">15929</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="n">f</span><span class="w"> </span><span class="mi">2019</span><span class="o">-</span><span class="mi">04</span><span class="o">-</span><span class="mi">03</span><span class="w"> </span><span class="mi">11</span><span class="p">:</span><span class="mi">49</span><span class="p">:</span><span class="mi">21</span><span class="p">.</span><span class="mi">810564</span><span class="w"> </span><span class="mi">2019</span><span class="o">-</span><span class="mi">04</span><span class="o">-</span><span class="mi">03</span><span class="w"> </span><span class="mi">11</span><span class="p">:</span><span class="mi">49</span><span class="p">:</span><span class="mi">21</span><span class="p">.</span><span class="mi">810564</span>
</pre></div>
<p>Here, the <span class="caps">ID</span> i was looking for is in the second column, so i used the magical ViM regex
<tt class="docutils literal"><span class="pre">^[0-9]\+\t\(15929\|15941\)</span></tt> (but with all the other IDs in the parentheses, too), and again
changed the new IDs to the old ones.</p>
<p>I still can’t believe i didn’t mess up the database too much, but it loaded just fine, and the
<tt class="docutils literal">db:migrate</tt> command executet just fine.</p>
<p>I also had some problems with so nginx reverse proxying, but it all turned out well, and my
instance is back online after a year. Come follow me at <a class="reference external" href="https://social.polonkai.eu/@gergely">@gergely@polonkai.eu</a>!</p>
Relying on my ears to keep my posture2020-03-25T05:48:00+00:002020-03-25T05:48:00+00:00Gergely Polonkaitag:None,2020-03-25:blog/relying-on-my-ears-to-keep-my-posture/<p><strong><span class="caps">TL</span>;<span class="caps">DR</span></strong>: my ears helped me create an almost perfect pomodoro timer out of my own body.</p>
<p>There’s an ongoing pandemic for a few weeks now, so whoever can should and does stay at home.
Luckily we can do it at Benchmark.games, as we only used our office …</p><p><strong><span class="caps">TL</span>;<span class="caps">DR</span></strong>: my ears helped me create an almost perfect pomodoro timer out of my own body.</p>
<p>There’s an ongoing pandemic for a few weeks now, so whoever can should and does stay at home.
Luckily we can do it at Benchmark.games, as we only used our office to stay in physical contact
with each other because we are (somewhat) social creatures. But now we try to take care of each
other and even the bravest work from home now. I hope the last one who left the office turned off
the heating and the lights.</p>
<p>I’m also lucky because i have both a supporting family and a separate room in our house which we
didn’t really use until today. It’s so separate that its door opens to our front yard, not inside
the house. There is/was a lot of junk there, waiting to be sorted or to be thrown out, so i took
a day off and tidied it up a bit so i could put a table and a chair inside. I also did some
cabling work and now it has Internet connection, too. It’s not overly comfy, but it will do for
the few weeks/months this pandemic will (hopefully) take. I also found a lot of treasures i won’t
list here, except a set of 5.1 speakers.</p>
<p>After setting up my workstation i put the speakers around me. They are not ideally far from me
(the room is not big enough for that) but it will suffice. The front centre speaker is above my
head so the speakers and my head form an almost perfect tetrahedron. The back speakers are also
at the same distance behind me. I have a bad habit of leaning closer to the screen when i overly
focus on something, and with this setup i just realised that whenever i do it i get outside of the
centre zone of the speakers and music doesn’t <em>sound</em> right. This way i instantly stop leaning
closer to the screen and fix my posture. Also, since the front centre speaker is above me, if my
back starts to get tired and i start to stoop, my head gets lower, and once again the music
doesn’t sound right. I try to take mental notes whenever this happens and i realised that after
an hour it happens more often, signalling that i should take a break.</p>
<p>During my breaks i drink some water, visit the toilet, do some exercises, have launch, or whatever
fits me, but most importantly, i try not to think of my current work task (unless i was in the
middle of focused work, which is rare after around an hour). This helps me clear my mind so in
the next hour i can focus better on my next task.</p>
Chatting on Matrix/Riot with end to end encryption from within Emacs2020-03-09T11:48:00+00:002020-03-09T11:48:00+00:00Gergely Polonkaitag:None,2020-03-09:2020/03/09/chatting-on-matrix-with-e2ee-from-emacs/<p>Iʼm a happy user and administrator of a <a class="reference external" href="https://matrix.org">Matrix</a> instance. Itʼs a pretty
lonely one (iʼm the sole user of it) but i faced a lot of pros and cons of it in the last few
years. In case you havenʼt heard about Matrix yet, itʼs a messaging framework which …</p><p>Iʼm a happy user and administrator of a <a class="reference external" href="https://matrix.org">Matrix</a> instance. Itʼs a pretty
lonely one (iʼm the sole user of it) but i faced a lot of pros and cons of it in the last few
years. In case you havenʼt heard about Matrix yet, itʼs a messaging framework which, for a few
years now, supports end to end encryption (<span class="caps">E2EE</span>). It also serves as the base of a federated chat
service you might know as <a class="reference external" href="https://riot.im">Riot</a>.</p>
<p>Another thing iʼm also fond of is Emacs. Itʼs a great little <span class="caps">UI</span> for pretty much everything i do,
from note taking through managing email and calendar to software development. It even has a
Matrix client, <a class="reference external" href="https://github.com/alphapapa/matrix-client.el">matrix-client.el</a>, written by
well-known (in Emacs circles at least) <a class="reference external" href="https://whatthefuck.computer/">Ryan Rix</a> (rrix) and
<a class="reference external" href="https://github.com/alphapapa">alphapapa</a>. I tried it several times, but at the end i always
came back to the official Riot client because we use Emacs <span class="caps">E2EE</span> both at my company and in private
with some friends, and unfortunately <tt class="docutils literal"><span class="pre">matrix-client.el</span></tt> doesnʼt support it.</p>
<p>As this is the case with many other clients, like Fractal, some awesome folks of the Matrix
community wrote <a class="reference external" href="https://github.com/matrix-org/pantalaimon">Pantalaimon</a> to solve this problem.
It’s a Matrix proxy that can do <span class="caps">E2EE</span> for clients that donʼt support it natively.</p>
<div class="section" id="pantalaimon-1">
<h2>Pantalaimon</h2>
<p>Setting up Pantalaimon is really easy: you <tt class="docutils literal">pip install</tt> it, write a 4 lines long config fire, and
you are ready to go. Just make sure you have a valid certificate if you use <span class="caps">HTTPS</span>; Pantalaimon
wonʼt connect if not, and will communicate this problem poorly.</p>
<pre class="code dosini literal-block">
<span class="k">[example-matrix]</span><span class="w">
</span><span class="na">Homeserver</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">https://example.org/</span><span class="w">
</span><span class="na">ListenAddress</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">127.0.0.1</span><span class="w">
</span><span class="na">ListenPort</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">8765</span>
</pre>
<p>You simply start it with <tt class="docutils literal">pantalaimon</tt> or, if you are a systemd fan user, you can use the
<a class="reference external" href="https://github.com/matrix-org/pantalaimon/blob/master/contrib/pantalaimon.service">service file from their repository</a> with a
slight change in the path.</p>
</div>
<div class="section" id="matrix-client-el-1">
<h2>matrix-client.el</h2>
<p>Coming up next, the Emacs Matrix client.</p>
<p>The fastest route for me was to add <tt class="docutils literal">quelpa</tt> and <tt class="docutils literal"><span class="pre">quelpa-use-package</span></tt> to my config, and
install matrix-client.el using those. The only tricky part is that iʼm lazy, so i have
<tt class="docutils literal"><span class="pre">use-package-always-ensure</span></tt> enabled which doesnʼt play nice with Quelpa installed packages that
are not otherwise available on <span class="caps">MELPA</span> (or any other repository you might have enabled), hence the
<tt class="docutils literal">:ensure nil</tt> clause.</p>
<pre class="code common-lisp literal-block">
<span class="c1">;; Don’t use it unless you are lazy enough to do extra work later</span><span class="w">
</span><span class="p">(</span><span class="nv">customize-set-variable</span><span class="w"> </span><span class="ss">'use-package-always-ensure</span><span class="w"> </span><span class="no">t</span><span class="p">)</span><span class="w">
</span><span class="p">(</span><span class="nb">use-package</span><span class="w"> </span><span class="nv">quelpa</span><span class="p">)</span><span class="w">
</span><span class="p">(</span><span class="nb">use-package</span><span class="w"> </span><span class="nv">quelpa-use-package</span><span class="w">
</span><span class="ss">:after</span><span class="w"> </span><span class="nv">quelpa</span><span class="p">)</span><span class="w">
</span><span class="p">(</span><span class="nb">use-package</span><span class="w"> </span><span class="nv">matrix-client</span><span class="w">
</span><span class="ss">:after</span><span class="w"> </span><span class="nv">quelpa-use-package</span><span class="w">
</span><span class="ss">:ensure</span><span class="w"> </span><span class="no">nil</span><span class="w"> </span><span class="c1">;; you only need this if you have `use-package-always-ensure' set to non-nil</span><span class="w">
</span><span class="ss">:quelpa</span><span class="w"> </span><span class="p">(</span><span class="nv">matrix-client</span><span class="w"> </span><span class="ss">:fetcher</span><span class="w"> </span><span class="nv">github</span><span class="w"> </span><span class="ss">:repo</span><span class="w"> </span><span class="s">"alphapapa/matrix-client.el"</span><span class="w">
</span><span class="ss">:files</span><span class="w"> </span><span class="p">(</span><span class="ss">:defaults</span><span class="w"> </span><span class="s">"logo.png"</span><span class="w"> </span><span class="s">"matrix-client-standalone.el.sh"</span><span class="p">)))</span>
</pre>
</div>
<div class="section" id="run-it">
<h2>Run it!</h2>
<p>At this point you are ready to go. In Emacs, run <kbd>M-x matrix-client-frame</kbd> or use the
standalone script <tt class="docutils literal"><span class="pre">$<span class="caps">HOME</span>/.emacs.d/elpa/matrix-client-<span class="caps">VERSION</span>/matrix-client-standalone.sh</span></tt> to run
it in a separate process (might be a better choice if your server is lagging a lot).</p>
<p>When <tt class="docutils literal"><span class="pre">matrix-client.el</span></tt> is asking for your credentials, you just enter them: your User <span class="caps">ID</span> (<span class="caps">MXID</span>,
looks like <tt class="docutils literal">@user:example.org</tt>) and your password is as usual. Your server should be the one
you specified in your Pantalaimon config as the listening <span class="caps">IP</span> and port, so in my example it would
be <tt class="docutils literal"><span class="pre">http://localhost:8009</span></tt>. If you donʼt need <span class="caps">E2EE</span>, then you can simply enter your homeserverʼs
address here (and thus you donʼt need Pantalaimon).</p>
<p>You are all set! Happy (secretive) chatting, enjoy doing everything from your favourite <span class="caps">OS</span>, Emacs!</p>
</div>
Moving to Pelican2019-11-05T07:05:00+00:002019-11-05T07:05:00+00:00Gergely Polonkaitag:None,2019-11-05:moving-to-pelican.html<p>I spent the last few days with moving this site from Jekyll to Pelican.</p>
<div class="section" id="story-time">
<h2>Story time!</h2>
<p>It wasn’t just a plain conversion, though. I also added a lot of my old stories, previously
published on Medium (i will delete that account within a few days, so it doesn’t …</p></div><p>I spent the last few days with moving this site from Jekyll to Pelican.</p>
<div class="section" id="story-time">
<h2>Story time!</h2>
<p>It wasn’t just a plain conversion, though. I also added a lot of my old stories, previously
published on Medium (i will delete that account within a few days, so it doesn’t worth linking.)
These stories are published with their original date (i wrote most of theme long before Medium existed).</p>
</div>
<div class="section" id="why">
<h2>Why?</h2>
<p>I left GitHub Pages for about a year or so now to host my site for myself. (Remember when i <a class="reference external" href="blog/2015/4/25/good-bye-digital-ocean-hello-again-github.html">said
goodbye to DigitalOcean</a>? Well, it seems
this goodbye was not forever…) My reasons were mainly about privacy, and the urge to do it
myself; i probably won’t overcome this latter one, like, ever. I want to make my hands dirty with
stuff like this. I also believe in the web as the web, not as a centralised… thing.</p>
<p>Another reason was that i want to host this site not only on <span class="caps">HTTP</span>, but also on <a class="reference external" href="dat://gergely.polonkai.eu"><span class="caps">DAT</span></a> (there’s also a <a class="reference external" href="dat://f261">hash link</a>) and <span class="caps">IPFS</span>. It’s not impossible with
Jekyll, it’s just easier for me because of Python.</p>
<p>And thus, we arrive to my final reason. Pelican may not have as many users as Jekyll does, but
it’s written in Python. It’s a big plus for me as i work with Python every day. It’s easy for me
to hack on the engine itself if i have to (and i had to several times during the migration).</p>
</div>
<div class="section" id="where-did-the-comments-go">
<h2>Where did the comments go?</h2>
<p>My site is a static site. It means there’s no dynamic web engine behind it (it wouldn’n actually
work on <span class="caps">DAT</span>/<span class="caps">IPFS</span>, would it?) It is so for a long while now. As such, i used Disqus for comments
but, again for privacy reasons, i have removed it completely. Also, i didn’t have comments anyway.</p>
</div>
<div class="section" id="what-happened-with-the-looks">
<h2>What happened with the looks?</h2>
<p>I didn’t convert my previous theme to Pelican format. I might, in the future, as i’m not
completely happy with the new layout. I like the new fonts, though, so maybe it will be a healthy
merger of the two. The reason for not doing it instantly is because the old site relied heavily
on external JavaScript and <span class="caps">CSS</span> files (like Bootstrap) which doesn’t work well with Dat and <span class="caps">IPFS</span>
(yes, most Dat browsers are capable of opening them from <span class="caps">HTTP</span>, but then what’s the point of the
whole conversion?) Now anyone can browse my site without an internet connection, and enjoy it in
its full glory.</p>
</div>
<div class="section" id="what-is-left">
<h2>What is left?</h2>
<p>I still want to do a lot of <span class="caps">CSS</span> tweaks, maybe converting the whole <span class="caps">CSS</span> part to <span class="caps">SCSS</span>/<span class="caps">SASS</span> (i
suspect the built-in theme of Pelican was written in <span class="caps">SASS</span>, i just need to find the source).
Nothing big, just to clean up the code.</p>
<p>I also want to make sure my site renders well with screen readers. I don’t have much content, but
i want it to make accessible for everyone. So keep an eye on the site, it might change soon!</p>
</div>
Check if the last Git commit has test coverage2018-07-26T12:49:52+00:002018-07-26T12:49:52+00:00Gergely Polonkaitag:None,2018-07-26:2018/07/26/check-if-last-git-commit-has-test-coverage/<p>I use Python at work and for private projects. I also aim to write tests for my code, especially
recently. And as I usually don’t start from 100% code coverage (<span class="caps">TDD</span> is not my game), I at least
want to know if the code I just wrote have full …</p><p>I use Python at work and for private projects. I also aim to write tests for my code, especially
recently. And as I usually don’t start from 100% code coverage (<span class="caps">TDD</span> is not my game), I at least
want to know if the code I just wrote have full coverage.</p>
<p>The trick is to collect all the lines that changed, and all the lines that has no coverage. Then
compare the two, and you have the uncovered lines that changed!</p>
<div class="section" id="getting-the-list-of-changed-lines">
<h2>Getting the list of changed lines</h2>
<p>Recently, I bumped into
<a class="reference external" href="https://adam.younglogic.com/2018/07/testing-patch-has-test/">this article</a>. It is a great awk
script that lists the lines that changed in the latest commit. I have really no problem with awk,
but I’m pretty sure it can be done in Python, as that is my main language nowadays.</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">get_changed_lines</span><span class="p">():</span>
<span class="w"> </span><span class="sd">"""Get the line numbers that changed in the last commit</span>
<span class="sd"> """</span>
<span class="n">git_output</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">check_output</span><span class="p">(</span><span class="s1">'git show'</span><span class="p">,</span> <span class="n">shell</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">'utf-8'</span><span class="p">)</span>
<span class="n">current_file</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">lines</span> <span class="o">=</span> <span class="p">{}</span>
<span class="n">left</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">right</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">git_output</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">'</span><span class="se">\n</span><span class="s1">'</span><span class="p">):</span>
<span class="n">match</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">match</span><span class="p">(</span><span class="sa">r</span><span class="s1">'^@@ -([0-9]+),[0-9]+ [+]([0-9]+),[0-9]+ @@'</span><span class="p">,</span> <span class="n">line</span><span class="p">)</span>
<span class="k">if</span> <span class="n">match</span><span class="p">:</span>
<span class="n">left</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">match</span><span class="o">.</span><span class="n">groups</span><span class="p">()[</span><span class="mi">0</span><span class="p">])</span>
<span class="n">right</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">match</span><span class="o">.</span><span class="n">groups</span><span class="p">()[</span><span class="mi">1</span><span class="p">])</span>
<span class="k">continue</span>
<span class="k">if</span> <span class="n">re</span><span class="o">.</span><span class="n">match</span><span class="p">(</span><span class="sa">r</span><span class="s1">'^\+\+\+'</span><span class="p">,</span> <span class="n">line</span><span class="p">):</span>
<span class="n">current_file</span> <span class="o">=</span> <span class="n">line</span><span class="p">[</span><span class="mi">6</span><span class="p">:]</span>
<span class="k">continue</span>
<span class="k">if</span> <span class="n">re</span><span class="o">.</span><span class="n">match</span><span class="p">(</span><span class="sa">r</span><span class="s1">'^-'</span><span class="p">,</span> <span class="n">line</span><span class="p">):</span>
<span class="n">left</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="k">continue</span>
<span class="k">if</span> <span class="n">re</span><span class="o">.</span><span class="n">match</span><span class="p">(</span><span class="sa">r</span><span class="s1">'^[+]'</span><span class="p">,</span> <span class="n">line</span><span class="p">):</span>
<span class="c1"># Save this line number as changed</span>
<span class="n">lines</span><span class="o">.</span><span class="n">setdefault</span><span class="p">(</span><span class="n">current_file</span><span class="p">,</span> <span class="p">[])</span>
<span class="n">lines</span><span class="p">[</span><span class="n">current_file</span><span class="p">]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">right</span><span class="p">)</span>
<span class="n">right</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="k">continue</span>
<span class="n">left</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="n">right</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="k">return</span> <span class="n">lines</span>
</pre></div>
<p><span class="caps">OK</span>, not as short as the awk script, but works just fine.</p>
</div>
<div class="section" id="getting-the-uncovered-lines">
<h2>Getting the uncovered lines</h2>
<p>Coverage.py can list the uncovered lines with <tt class="docutils literal">coverage report <span class="pre">--show-missing</span></tt>. For
Calendar.social, this looks something like this:</p>
<div class="highlight"><pre><span></span>Name Stmts Miss Cover Missing
----------------------------------------------------------------------
calsocial/__init__.py 173 62 64% 44, 138-148, 200, 239-253, 261-280, 288-295, 308-309, 324-346, 354-363
calsocial/__main__.py 3 3 0% 4-9
calsocial/account.py 108 51 53% 85-97, 105-112, 125, 130-137, 148-160, 169-175, 184-200, 209-212, 221-234
calsocial/app_state.py 10 0 100%
calsocial/cache.py 73 11 85% 65-70, 98, 113, 124, 137, 156-159
calsocial/calendar_system/__init__.py 10 3 70% 32, 41, 48
calsocial/calendar_system/gregorian.py 77 0 100%
calsocial/config_development.py 11 11 0% 4-17
calsocial/config_testing.py 12 0 100%
calsocial/forms.py 198 83 58% 49, 59, 90, 136-146, 153, 161-169, 188-195, 198-206, 209-212, 228-232, 238-244, 252-253, 263-267, 273-277, 317-336, 339-342, 352-354, 362-374, 401-413
calsocial/models.py 372 92 75% 49-51, 103-106, 177, 180-188, 191-200, 203, 242-248, 257-268, 289, 307, 349, 352-359, 378, 392, 404-409, 416, 444, 447, 492-496, 503, 510, 516, 522, 525, 528, 535-537, 545-551, 572, 606-617, 620, 652, 655, 660, 700, 746-748, 762-767, 774-783, 899, 929, 932
calsocial/security.py 15 3 80% 36, 56-58
calsocial/utils.py 42 5 88% 45-48, 52-53
----------------------------------------------------------------------
TOTAL 1104 324 71%
</pre></div>
<p>All we have to do is converting these ranges into a list of numbers, and compare it with the
result of the previous function:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">get_uncovered_lines</span><span class="p">(</span><span class="n">changed_lines</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Get the full list of lines that has not been covered by tests</span>
<span class="sd"> """</span>
<span class="n">column_widths</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">uncovered_lines</span> <span class="o">=</span> <span class="p">{}</span>
<span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">sys</span><span class="o">.</span><span class="n">stdin</span><span class="p">:</span>
<span class="n">line</span> <span class="o">=</span> <span class="n">line</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span>
<span class="k">if</span> <span class="n">line</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">'---'</span><span class="p">):</span>
<span class="k">continue</span>
<span class="k">if</span> <span class="n">line</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">'Name '</span><span class="p">):</span>
<span class="n">match</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">match</span><span class="p">(</span><span class="sa">r</span><span class="s1">'^(Name +)(Stmts +)(Miss +)(Cover +)Missing$'</span><span class="p">,</span> <span class="n">line</span><span class="p">)</span>
<span class="k">assert</span> <span class="n">match</span>
<span class="n">column_widths</span> <span class="o">=</span> <span class="p">[</span><span class="nb">len</span><span class="p">(</span><span class="n">col</span><span class="p">)</span> <span class="k">for</span> <span class="n">col</span> <span class="ow">in</span> <span class="n">match</span><span class="o">.</span><span class="n">groups</span><span class="p">()]</span>
<span class="k">continue</span>
<span class="n">name</span> <span class="o">=</span> <span class="p">[</span>
<span class="n">line</span><span class="p">[</span><span class="nb">sum</span><span class="p">(</span><span class="n">column_widths</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="n">idx</span><span class="p">]):</span><span class="nb">sum</span><span class="p">(</span><span class="n">column_widths</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="n">idx</span><span class="p">])</span> <span class="o">+</span> <span class="n">width</span><span class="p">]</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span>
<span class="k">for</span> <span class="n">idx</span><span class="p">,</span> <span class="n">width</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">column_widths</span><span class="p">)][</span><span class="mi">0</span><span class="p">]</span>
<span class="n">missing</span> <span class="o">=</span> <span class="n">line</span><span class="p">[</span><span class="nb">sum</span><span class="p">(</span><span class="n">column_widths</span><span class="p">):]</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span>
<span class="k">for</span> <span class="n">value</span> <span class="ow">in</span> <span class="n">missing</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">', '</span><span class="p">):</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">value</span><span class="p">:</span>
<span class="k">continue</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">number</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
<span class="k">except</span> <span class="ne">ValueError</span><span class="p">:</span>
<span class="n">first</span><span class="p">,</span> <span class="n">last</span> <span class="o">=</span> <span class="n">value</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">'-'</span><span class="p">)</span>
<span class="n">lines</span> <span class="o">=</span> <span class="nb">range</span><span class="p">(</span><span class="nb">int</span><span class="p">(</span><span class="n">first</span><span class="p">),</span> <span class="nb">int</span><span class="p">(</span><span class="n">last</span><span class="p">)</span> <span class="o">+</span> <span class="mi">1</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">lines</span> <span class="o">=</span> <span class="nb">range</span><span class="p">(</span><span class="n">number</span><span class="p">,</span> <span class="n">number</span> <span class="o">+</span> <span class="mi">1</span><span class="p">)</span>
<span class="k">for</span> <span class="n">lineno</span> <span class="ow">in</span> <span class="n">lines</span><span class="p">:</span>
<span class="k">if</span> <span class="n">name</span> <span class="ow">in</span> <span class="n">changed_lines</span> <span class="ow">and</span> <span class="n">lineno</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">changed_lines</span><span class="p">[</span><span class="n">name</span><span class="p">]:</span>
<span class="n">uncovered_lines</span><span class="o">.</span><span class="n">setdefault</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="p">[])</span>
<span class="n">uncovered_lines</span><span class="p">[</span><span class="n">name</span><span class="p">]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">lineno</span><span class="p">)</span>
<span class="k">return</span> <span class="n">uncovered_lines</span>
</pre></div>
<p>At the end we have a dictionary that has filenames as keys, and a list of changed but uncovered lines.</p>
</div>
<div class="section" id="converting-back-to-ranges">
<h2>Converting back to ranges</h2>
<p>To make the final result more readable, let’s convert them back to a nice <tt class="docutils literal"><span class="pre">from_line-to_line</span></tt>
range list first:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">line_numbers_to_ranges</span><span class="p">():</span>
<span class="w"> </span><span class="sd">"""List the lines that has not been covered</span>
<span class="sd"> """</span>
<span class="n">changed_lines</span> <span class="o">=</span> <span class="n">get_changed_lines</span><span class="p">()</span>
<span class="n">uncovered_lines</span> <span class="o">=</span> <span class="n">get_uncovered_lines</span><span class="p">(</span><span class="n">changed_lines</span><span class="p">)</span>
<span class="n">line_list</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">filename</span><span class="p">,</span> <span class="n">lines</span> <span class="ow">in</span> <span class="n">uncovered_lines</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
<span class="n">lines</span> <span class="o">=</span> <span class="nb">sorted</span><span class="p">(</span><span class="n">lines</span><span class="p">)</span>
<span class="n">last_value</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">ranges</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">lineno</span> <span class="ow">in</span> <span class="n">lines</span><span class="p">:</span>
<span class="k">if</span> <span class="n">last_value</span> <span class="ow">and</span> <span class="n">last_value</span> <span class="o">+</span> <span class="mi">1</span> <span class="o">==</span> <span class="n">lineno</span><span class="p">:</span>
<span class="n">ranges</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">lineno</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">ranges</span><span class="o">.</span><span class="n">append</span><span class="p">([</span><span class="n">lineno</span><span class="p">])</span>
<span class="n">last_value</span> <span class="o">=</span> <span class="n">lineno</span>
<span class="n">range_list</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">range_</span> <span class="ow">in</span> <span class="n">ranges</span><span class="p">:</span>
<span class="n">first</span> <span class="o">=</span> <span class="n">range_</span><span class="o">.</span><span class="n">pop</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
<span class="k">if</span> <span class="n">range_</span><span class="p">:</span>
<span class="n">range_list</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="sa">f</span><span class="s1">'</span><span class="si">{</span><span class="n">first</span><span class="si">}</span><span class="s1">-</span><span class="si">{</span><span class="n">range_</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span><span class="si">}</span><span class="s1">'</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">range_list</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">first</span><span class="p">))</span>
<span class="n">line_list</span><span class="o">.</span><span class="n">append</span><span class="p">((</span><span class="n">filename</span><span class="p">,</span> <span class="s1">', '</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">range_list</span><span class="p">)))</span>
<span class="k">return</span> <span class="n">line_list</span>
</pre></div>
</div>
<div class="section" id="printing-the-result">
<h2>Printing the result</h2>
<p>Now all that is left is to print the result on the screen in a format digestable by a human being:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">tabular_print</span><span class="p">(</span><span class="n">uncovered_lines</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Print the list of uncovered lines on the screen in a tabbed format</span>
<span class="sd"> """</span>
<span class="n">max_filename_len</span> <span class="o">=</span> <span class="nb">max</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span> <span class="k">for</span> <span class="n">data</span> <span class="ow">in</span> <span class="n">uncovered_lines</span><span class="p">)</span>
<span class="k">for</span> <span class="n">filename</span><span class="p">,</span> <span class="n">lines</span> <span class="ow">in</span> <span class="n">uncovered_lines</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="n">filename</span><span class="o">.</span><span class="n">ljust</span><span class="p">(</span><span class="n">max_filename_len</span> <span class="o">+</span> <span class="mi">2</span><span class="p">)</span> <span class="o">+</span> <span class="n">lines</span><span class="p">)</span>
</pre></div>
<p>And we are done.</p>
</div>
<div class="section" id="conclusion">
<h2>Conclusion</h2>
<p>This task never seemed hard to accomplish, but somehow I never put enough energy into it to make
it happen. Kudos to Adam Young doing some legwork for me!</p>
</div>
Recurring events are hard2018-07-19T13:22:00+00:002018-07-19T13:22:00+00:00Gergely Polonkaitag:None,2018-07-19:2018/07/19/recurring-events/<p>It was almost a month ago when I <a class="reference external" href="2018/06/26/please-welcome-calendar-social/">announced</a> the development of Calendar.social.
Since then I’m over some interesting and some less interesting stuff; (web) development, after
all, is just a recurrence of patterns. Speaking of recurrence, I arrived to a really interesting
topic: recurring events.</p>
<p>My initial …</p><p>It was almost a month ago when I <a class="reference external" href="2018/06/26/please-welcome-calendar-social/">announced</a> the development of Calendar.social.
Since then I’m over some interesting and some less interesting stuff; (web) development, after
all, is just a recurrence of patterns. Speaking of recurrence, I arrived to a really interesting
topic: recurring events.</p>
<p>My initial thought was like “oh, that’s easy! Let’s insert all future occurences as a separate
<tt class="docutils literal">Event</tt> object, linking to the original one for the details. That makes handling exceptions
easy, as I just have to update/delete that specific instance.” Well, not really. I mean, an
event repeating daily <em>forever</em> would fill up the database quickly, isn’t it? That’s when I
decided to look how other projects do it.</p>
<p>As it turns out, my first thought is about the same as everyone else has their mind, with about
the same reasons. Then, they usually turn down the idea just like I did. And instead, they
implement recurrence patterns and exception patterns.</p>
<p>My favourite is <a class="reference external" href="https://github.com/bmoeskau/Extensible/blob/master/recurrence-overview.md">this article</a> so far. The author
suggests to use the recurrence patterns specced by <a class="reference external" href="http://www.ietf.org/rfc/rfc2445.txt"><span class="caps">RFC2445</span></a> (the spec for the iCalendar format). The interesting part
in this solution is how to query recurring events: you simply store the timestamp of the last
occurence of the events (or, if the event repeats forever, the greatest timestamp your database supports.)</p>
<p>Choosing the maximum date seemed to be the tricky one, but it turned out both Python and popular
<span class="caps">SQL</span> backends support dates up to the end of year 9999.</p>
Please welcome Calendar.social2018-06-26T05:36:00+00:002018-06-26T05:36:00+00:00Gergely Polonkaitag:None,2018-06-26:2018/06/26/please-welcome-calendar-social/<p>I started looking at decentralised/federated tools some years ago, but other than Matrix I didn’t
use any of them until recently. Then this February I joined the Fediverse (federated universe) by
spinning up my own <a class="reference external" href="https://joinmastodon.org/">Mastodon</a> instance. I’m not going to lie, this
place is pure awesome …</p><p>I started looking at decentralised/federated tools some years ago, but other than Matrix I didn’t
use any of them until recently. Then this February I joined the Fediverse (federated universe) by
spinning up my own <a class="reference external" href="https://joinmastodon.org/">Mastodon</a> instance. I’m not going to lie, this
place is pure awesome. I follow only 53 people but unlike on Twitter or Facebook, I can’t just
scroll through my timeline; I have to read them all. These 53 accounts are real persons over the
Internet with meaningful posts. I could never find this on the noisy Twitter or Facebook timeline
during the last 10 or so years.</p>
<p>Bragging aside, and given my strive for learning, I wanted to give back something to this
community. I thought about an image sharing platform where people can share their photo albums
with each other, but I realised I’m not that good at image handling. So I decided to make a
calendar instead.</p>
<p>My app, creatively codenamed Calendar.social, aims to be a calendar service similar to Google
Calendar (and, obviously, any calendar app you can find out there) but instead of using emails, it
will use ActivityPub to share all the details (although I might add e-mail support sooner or later.)</p>
<p>I have a <em>lot</em> on my mind about what this tool should/could do when it’s done. In no particular
order, here’s a list of them:</p>
<ul class="simple">
<li>events that can be private (only you and the (optional) guests see them), or public (anyone can
see them). They will have all the usual fields like start/end time, location, and maybe an icon
and a cover photo</li>
<li>multiple calendars you would expect from a calendar app. This way you can separate your
birthday reminders from the dentist appointments</li>
<li>event sharing over ActivityPub and other channels (to be decided, but I think you can take email
and maybe Matrix for granted.)</li>
<li>full calendar sharing. The other party can get access from a very basic free/busy level to full
write access (which might be a good idea for family or company wide calendars.)</li>
<li>Holiday calendars that store national/bank holidays. Users can subscribe to them to see the
holidays of a given country/area, and optionally set them as busy (on holiday weekdays) or free
(on weekends that are actually workdays for some reason.)</li>
<li>Reminders! Because you obviously don’t want to forget the birthday of your significant other,
your anniversary, or your barber appointment.</li>
<li>All this developed with time zones, localisation, and accessibility in mind.</li>
</ul>
<p>That, and anything more that comes to my mind.</p>
<p>You can follow the development <a class="reference external" href="https://gitea.polonkai.eu/gergely/calendar-social">here</a>. Also,
feel free to ping me with your ideas on my <a class="reference external" href="https://social.polonkai.eu/@gergely">Mastodon account</a>, <a class="reference external" href="https://matrix.to/#/@gergely:polonkai.eu">Matrix</a>, or any other channels you can find under the
“Contact me” menu.</p>
Add SysAdmin day to Emacs Calendar2017-10-02T09:37:52+00:002017-10-02T09:37:52+00:00Gergely Polonkaitag:None,2017-10-02:2017/10/02/add-sysadmin-day-to-emacs-calendar/<p>I’m a SysAdmin since 1998. Maybe a bit earlier, if you count managing our home computer. This
means <a class="reference external" href="http://sysadminday.com/">SysAdmin Day</a> is also celebrating me. However, my Emacs
Calendar doesn’t show it for some reason.</p>
<p>The solution is pretty easy:</p>
<div class="highlight"><pre><span></span><span class="p">(</span><span class="nv">add-to-list</span><span class="w"> </span><span class="ss">'holiday-other-holidays</span><span class="w"> </span><span class="o">'</span><span class="p">(</span><span class="nv">holiday-float</span><span class="w"> </span><span class="mi">7</span><span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="mi">-1</span><span class="w"> </span><span class="s">"SysAdmin Day"</span><span class="p">)</span><span class="w"> </span><span class="no">t …</span></pre></div><p>I’m a SysAdmin since 1998. Maybe a bit earlier, if you count managing our home computer. This
means <a class="reference external" href="http://sysadminday.com/">SysAdmin Day</a> is also celebrating me. However, my Emacs
Calendar doesn’t show it for some reason.</p>
<p>The solution is pretty easy:</p>
<div class="highlight"><pre><span></span><span class="p">(</span><span class="nv">add-to-list</span><span class="w"> </span><span class="ss">'holiday-other-holidays</span><span class="w"> </span><span class="o">'</span><span class="p">(</span><span class="nv">holiday-float</span><span class="w"> </span><span class="mi">7</span><span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="mi">-1</span><span class="w"> </span><span class="s">"SysAdmin Day"</span><span class="p">)</span><span class="w"> </span><span class="no">t</span><span class="p">)</span>
</pre></div>
<p>Now invoke <kbd>M-x holidays-list</kbd> for any year, choosing “Other” as the category, and there you go:</p>
<div class="highlight"><pre><span></span>…
Friday, July 28, 2017: SysAdmin Day
…
</pre></div>
Category-based logging with Flask2017-03-26T20:00:52+00:002017-03-26T20:00:52+00:00Gergely Polonkaitag:None,2017-03-26:2017/03/27/category-based-logging-with-flask/<p>I’m in a team who are developing a Flask-based web application, which uses logging extensively.
For a while now it spews out a lot of lines so the need arose to index them in ElasticSearch, and
more importantly, to search through them for auditing purposes. This latter user story …</p><p>I’m in a team who are developing a Flask-based web application, which uses logging extensively.
For a while now it spews out a lot of lines so the need arose to index them in ElasticSearch, and
more importantly, to search through them for auditing purposes. This latter user story brought up
one more question: why don’t we categorize our log messages? I quickly came up with an extended
log format (<tt class="docutils literal">[auth]</tt> is the new category name):</p>
<div class="highlight"><pre><span></span>[2017-01-14 00:55:42,554] [8286] [INFO] [auth] invalid password for john@example.com [at __init__.py:12, in function utils.validate_login]
</pre></div>
<p>Here, <tt class="docutils literal">[auth]</tt> is the category name. In the ideal solution, all I’d have to
do is adding <tt class="docutils literal">%(category)s</tt> to my formatter, and I could call</p>
<div class="highlight"><pre><span></span><span class="n">app</span><span class="o">.</span><span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="n">message</span><span class="p">,</span> <span class="n">extra</span><span class="o">=</span><span class="p">{</span><span class="s1">'auth'</span><span class="p">:</span> <span class="s1">'invalid password'</span><span class="p">})</span>
</pre></div>
<p>to achieve this output. Unfortunately, <tt class="docutils literal">Flask.logger</tt> (and, in the background, the <tt class="docutils literal">logging</tt>
module) is not that easy to tame.</p>
<p>As it turns out, a Flask application’s <tt class="docutils literal">logger</tt> property is a subclass of <tt class="docutils literal">logging.Logger</tt>, so
my first idea was to monkey patch that class. When the app’s logger is initialised, it subclasses
<tt class="docutils literal">logging.Logger</tt> and tweaks the log level so it goes down to <tt class="docutils literal">logging.<span class="caps">DEBUG</span></tt> if the app is
running in debug mode. This is done by using a different logger class depending on the app
config. Fortunately it doesn’t directly subclass <tt class="docutils literal">logging.Logger</tt>; it calls
<tt class="docutils literal">logging.getLoggerClass()</tt> to find which class it should extend. To achieve my goals, all I had
to do is to subclass the original logger class, and pass it to <tt class="docutils literal">logging.setLoggerClass()</tt>
<em>before</em> I initialise my app, and I have a fail-safe(ish) solution. So far so good, on to the
extra category parameter.</p>
<p>Now if you add a new variable to the formatter like my new <tt class="docutils literal">%(category)s</tt>, you get a nifty
<tt class="docutils literal">KeyError</tt> saying there is no <tt class="docutils literal">category</tt> in the format expansion dictionary. If you add
<tt class="docutils literal"><span class="pre">category='auth'</span></tt> to the <tt class="docutils literal">app.logger.info()</tt> calls and its cousins, it’s fine, because these
methods use the magic <tt class="docutils literal">**kwarg</tt> argument to swallow it. Everything goes well until control
arrives to the <tt class="docutils literal">_log()</tt> method: it complains about that extra <tt class="docutils literal">category</tt> keyword argument.
Taking a peek at Python’s internals, I found two things: <tt class="docutils literal"><span class="pre">info()`,</span> `error()</tt>, and co. pass
<tt class="docutils literal">*args</tt> and <tt class="docutils literal">**kwargs</tt> to <tt class="docutils literal">_log()</tt> unmodified, and the <tt class="docutils literal">_log()</tt> method doesn’t have
<tt class="docutils literal">**kwargs</tt> present in its definition to swallow it. A little doc reading later I found that if
I want to pass extra arguments for such a formatter, I should do it via the <tt class="docutils literal">extra</tt> keyword
argument to <tt class="docutils literal">_log()</tt>. A call like <tt class="docutils literal"><span class="pre">app.logger.info('invalid</span> password', <span class="pre">extra={'category':</span>
<span class="pre">'auth'})</span></tt> solved the problem. Now <em>that</em> is tedious.</p>
<p>My first idea was to override all the standard logging methods like <tt class="docutils literal">info()</tt> and <tt class="docutils literal">error()</tt>,
and handle <tt class="docutils literal">category</tt> there. But this resulted in lots of repeating code. I changed the
specification a bit, so my calls would look like <tt class="docutils literal"><span class="pre">info('message',</span> <span class="pre">category='auth)</span></tt> instead of
the original plan of <tt class="docutils literal"><span class="pre">info('auth',</span> 'message')</tt>: as the logging methods pass all keyword
arguments to <tt class="docutils literal">_log()</tt>, I can handle it there. So at the end, my new logger class only patches
<tt class="docutils literal">_log()</tt>, by picking out <tt class="docutils literal">category</tt> from the kwarg list, and inserting it to <tt class="docutils literal">extra</tt> before
calling <tt class="docutils literal">super</tt>.</p>
<p>As you can see, this is a bit ugly solution. It requires me, the app
author, to know about Flask’s internals (that I can set my own logging class
before the app is created, and so the app will use it.) This means if Flask
developers change the way how logging is done, I have to adapt and find a
workaround for the new version (well, unless they let me directly set the
logging class. That would make it easy.)</p>
<p>What is worse, I must know about Python internals. I know the <tt class="docutils literal">extra</tt> kwarg is documented well
(I just failed to notice), but this made adding a new formatter variable so hard. Python version
doesn’t change as often as Flask version in this project, and I think the standard library won’t
really change until 4.0, so I don’t think my tampering with a “protected” method will cause any
trouble in the future. Still, this makes me feel a bit uneasy.</p>
<p>All the above can be circumvented if this class, and the whole solution have some tests. As my
class uses the same method as Flask (use <tt class="docutils literal">logging.getLoggerClass()</tt> as a base class instead of
using <tt class="docutils literal">logging.Logger()</tt> directly), if the base logger class changes in Python or in the running
environment, my app won’t care. By checking if the app logger can use my special <tt class="docutils literal">category</tt>
variable (ie. it doesn’t raise an exception <em>and</em> the category actually gets into the log output),
I made sure my class is used as a base in Flask, so if they change the way they construct
<tt class="docutils literal">app.logger</tt>, I will know about it when I first run my tests after upgrading Flask.</p>
<p>If you are interested in such functionality (and more), you can grab it from <a class="reference external" href="https://github.com/gergelypolonkai/flask-logging-extras">GitHub</a>, or via <a class="reference external" href="https://pypi.python.org/pypi/Flask-Logging-Extras/">PyPI</a>.</p>
Rename automatically named foreign keys with Alembic2017-01-02T09:41:23+00:002017-01-02T09:41:23+00:00Gergely Polonkaitag:None,2017-01-02:2017/01/02/rename-automatically-named-foreign-keys-with-alembic/<p>I have recently messed up my Alembic migrations while modifying my SQLAlchemy models. To start
with, I didn’t update the auto-generated migration files to name the indexes/foreign keys a name,
so Alembic used its own naming scheme. This is not an actual problem until you have to modify …</p><p>I have recently messed up my Alembic migrations while modifying my SQLAlchemy models. To start
with, I didn’t update the auto-generated migration files to name the indexes/foreign keys a name,
so Alembic used its own naming scheme. This is not an actual problem until you have to modify
columns that have such constraints. I have since fixed this problem, but first I had to find
which column references what (I had no indexes other than primary key back then, so I could go
with foreign keys only). Here is a query I put together, mostly using <a class="reference external" href="http://www.binarytides.com/list-foreign-keys-in-mysql/">this article</a>.</p>
<div class="highlight"><pre><span></span><span class="k">SELECT</span><span class="w"> </span><span class="k">constraint_name</span><span class="p">,</span>
<span class="w"> </span><span class="n">CONCAT</span><span class="p">(</span><span class="k">table_name</span><span class="p">,</span><span class="w"> </span><span class="s1">'.'</span><span class="p">,</span><span class="w"> </span><span class="k">column_name</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="s1">'foreign key'</span><span class="p">,</span>
<span class="w"> </span><span class="n">CONCAT</span><span class="p">(</span><span class="n">referenced_table_name</span><span class="p">,</span><span class="w"> </span><span class="s1">'.'</span><span class="p">,</span><span class="w"> </span><span class="n">referenced_column_name</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="s1">'references'</span>
<span class="k">FROM</span><span class="w"> </span><span class="n">information_schema</span><span class="p">.</span><span class="n">key_column_usage</span>
<span class="k">WHERE</span><span class="w"> </span><span class="n">referenced_table_name</span><span class="w"> </span><span class="k">IS</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="k">NULL</span><span class="w"> </span><span class="k">AND</span>
<span class="w"> </span><span class="n">table_schema</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'my_app'</span><span class="p">;</span>
</pre></div>
<p>Now I could easily drop such constraints using</p>
<div class="highlight"><pre><span></span><span class="n">alembic</span><span class="o">.</span><span class="n">op</span><span class="o">.</span><span class="n">drop_constraint</span><span class="p">(</span><span class="s1">'users_ibfk1'</span><span class="p">,</span> <span class="s1">'users'</span><span class="p">,</span> <span class="n">type_</span><span class="o">=</span><span class="s1">'foreignkey'</span><span class="p">)</span>
</pre></div>
<p>and recreate them with</p>
<div class="highlight"><pre><span></span><span class="n">alembic</span><span class="o">.</span><span class="n">op</span><span class="o">.</span><span class="n">create_foreign_key</span><span class="p">(</span><span class="s1">'fk_user_client'</span><span class="p">,</span> <span class="s1">'users'</span><span class="p">,</span> <span class="s1">'clients'</span><span class="p">,</span> <span class="p">[</span><span class="s1">'client_id'</span><span class="p">],</span> <span class="p">[</span><span class="s1">'id'</span><span class="p">])</span>
</pre></div>
Finding non-translated strings in Python code2016-12-22T09:35:11+00:002016-12-22T09:35:11+00:00Gergely Polonkaitag:None,2016-12-22:2016/12/22/finding-non-translated-strings-in-python-code/<p>When creating multilingual software, be it on the web, mobile, or desktop, you will eventually
fail to mark strings as translatable. I know, I know, we developers are superhuman and never do
that, but somehow I stopped trusting myself recently, so I came up with an idea.</p>
<p>Right now I …</p><p>When creating multilingual software, be it on the web, mobile, or desktop, you will eventually
fail to mark strings as translatable. I know, I know, we developers are superhuman and never do
that, but somehow I stopped trusting myself recently, so I came up with an idea.</p>
<p>Right now I assist in the creation of a multilingual site/web application, where a small part of
the strings come from the Python code instead of <span class="caps">HTML</span> templates. Call it bad practice if you
like, but I could not find a better way yet.</p>
<p>As a start, I tried to parse the source files with simple regular expressions, so I could find
anything between quotation marks or apostrophes. This attempt quickly failed with strings that
had such characters inside, escaped or not; my regexps became so complex I lost all hope. Then
the magic word “lexer” came to mind.</p>
<p>While searching for ready made Python lexers, I bumped into the awesome <tt class="docutils literal">ast</tt> module. <span class="caps">AST</span>
stands for Abstract Syntax Tree, and this module does that: parses a Python file and returns a
tree of nodes. For walking through these nodes there is a <tt class="docutils literal">NodeVisitor</tt> class (among other
means), which is meant to be subclassed. You add a bunch of <tt class="docutils literal">visitN</tt> methods (where <tt class="docutils literal">N</tt> is an
<tt class="docutils literal">ast</tt> class name like <tt class="docutils literal">Str</tt> or <tt class="docutils literal">Call</tt>), instantiate it, and call its <tt class="docutils literal">visit()</tt> method with
the root node. For example, the <tt class="docutils literal">visitStr()</tt> method will be invoked for every string it finds.</p>
<div class="section" id="how-does-it-work">
<h2>How does it work?</h2>
<p>Before getting into the details, let’s me present you the code I made:</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">ast</span>
<span class="kn">import</span> <span class="nn">gettext</span>
<span class="kn">from</span> <span class="nn">gettext</span> <span class="kn">import</span> <span class="n">gettext</span> <span class="k">as</span> <span class="n">_</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="k">def</span> <span class="nf">get_func_name</span><span class="p">(</span><span class="n">node</span><span class="p">):</span>
<span class="bp">cls</span> <span class="o">=</span> <span class="n">node</span><span class="o">.</span><span class="vm">__class__</span><span class="o">.</span><span class="vm">__name__</span>
<span class="k">if</span> <span class="bp">cls</span> <span class="o">==</span> <span class="s1">'Call'</span><span class="p">:</span>
<span class="k">return</span> <span class="n">get_func_name</span><span class="p">(</span><span class="n">node</span><span class="o">.</span><span class="n">func</span><span class="p">)</span>
<span class="k">elif</span> <span class="bp">cls</span> <span class="o">==</span> <span class="s1">'Attribute'</span><span class="p">:</span>
<span class="k">return</span> <span class="s1">'</span><span class="si">{}</span><span class="s1">.</span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
<span class="n">get_func_name</span><span class="p">(</span><span class="n">node</span><span class="o">.</span><span class="n">value</span><span class="p">),</span>
<span class="n">node</span><span class="o">.</span><span class="n">attr</span><span class="p">)</span>
<span class="k">elif</span> <span class="bp">cls</span> <span class="o">==</span> <span class="s1">'Name'</span><span class="p">:</span>
<span class="k">return</span> <span class="n">get_func_name</span><span class="p">(</span><span class="n">node</span><span class="o">.</span><span class="n">id</span><span class="p">)</span>
<span class="k">elif</span> <span class="bp">cls</span> <span class="o">==</span> <span class="s1">'str'</span><span class="p">:</span>
<span class="k">return</span> <span class="n">node</span>
<span class="k">elif</span> <span class="bp">cls</span> <span class="o">==</span> <span class="s1">'Str'</span><span class="p">:</span>
<span class="k">return</span> <span class="s2">"<String literal>"</span>
<span class="k">elif</span> <span class="bp">cls</span> <span class="o">==</span> <span class="s1">'Subscript'</span><span class="p">:</span>
<span class="k">return</span> <span class="s1">'</span><span class="si">{}</span><span class="s1">[</span><span class="si">{}</span><span class="s1">]'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">get_func_name</span><span class="p">(</span><span class="n">node</span><span class="o">.</span><span class="n">value</span><span class="p">),</span>
<span class="n">get_func_name</span><span class="p">(</span><span class="n">node</span><span class="o">.</span><span class="n">slice</span><span class="p">))</span>
<span class="k">elif</span> <span class="bp">cls</span> <span class="o">==</span> <span class="s1">'Index'</span><span class="p">:</span>
<span class="k">return</span> <span class="n">get_func_name</span><span class="p">(</span><span class="n">node</span><span class="o">.</span><span class="n">value</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">'ERROR: Unknown class: </span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">cls</span><span class="p">))</span>
<span class="k">class</span> <span class="nc">ShowStrings</span><span class="p">(</span><span class="n">ast</span><span class="o">.</span><span class="n">NodeVisitor</span><span class="p">):</span>
<span class="n">TRANSLATION_FUNCTIONS</span> <span class="o">=</span> <span class="p">[</span>
<span class="s1">'_'</span><span class="p">,</span> <span class="c1"># gettext.gettext is often imported under this name</span>
<span class="s1">'gettext'</span><span class="p">,</span>
<span class="s1">'gettext.gettext'</span><span class="p">,</span>
<span class="c1"># FIXME: this list is pretty much incomplete</span>
<span class="p">]</span>
<span class="n">UNTRANSLATED</span> <span class="o">=</span> <span class="s1">'untranslated 9'</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">filename</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="nb">super</span><span class="p">(</span><span class="n">ShowStrings</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">in_call</span> <span class="o">=</span> <span class="p">[]</span>
<span class="bp">self</span><span class="o">.</span><span class="n">filename</span> <span class="o">=</span> <span class="n">filename</span> <span class="ow">or</span> <span class="s1">'<parsed string>'</span>
<span class="k">def</span> <span class="nf">visit_with_trace</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">node</span><span class="p">,</span> <span class="n">func</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">in_call</span><span class="o">.</span><span class="n">append</span><span class="p">((</span><span class="n">func</span><span class="p">,</span> <span class="n">node</span><span class="o">.</span><span class="n">lineno</span><span class="p">,</span> <span class="n">node</span><span class="o">.</span><span class="n">col_offset</span><span class="p">))</span>
<span class="bp">self</span><span class="o">.</span><span class="n">visit</span><span class="p">(</span><span class="n">node</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">in_call</span><span class="o">.</span><span class="n">pop</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">visit_Str</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">node</span><span class="p">):</span>
<span class="c1"># TODO: make it possible to ignore untranslated strings</span>
<span class="c1"># TODO: make this ignore docstrings</span>
<span class="c1"># if we are not in a translator function, issue a warning</span>
<span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">in_call</span> <span class="ow">or</span> \
<span class="bp">self</span><span class="o">.</span><span class="n">in_call</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span> <span class="ow">not</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">TRANSLATION_FUNCTIONS</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">funcname</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">in_call</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span>
<span class="k">except</span> <span class="ne">IndexError</span><span class="p">:</span>
<span class="n">funcname</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">funcall_msg</span> <span class="o">=</span> <span class="s2">"outside a function call"</span> <span class="k">if</span> <span class="n">funcname</span> <span class="ow">is</span> <span class="kc">None</span> \
<span class="k">else</span> <span class="s2">"inside a call to </span><span class="si">{funcname}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
<span class="n">funcname</span><span class="o">=</span><span class="n">funcname</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">"WARNING: Untranslated string found at "</span>
<span class="s2">"</span><span class="si">{filename}</span><span class="s2">:</span><span class="si">{line}</span><span class="s2">:</span><span class="si">{col}</span><span class="s2"> </span><span class="si">{funcall_msg}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
<span class="n">filename</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">filename</span><span class="p">,</span>
<span class="n">line</span><span class="o">=</span><span class="n">node</span><span class="o">.</span><span class="n">lineno</span><span class="p">,</span>
<span class="n">col</span><span class="o">=</span><span class="n">node</span><span class="o">.</span><span class="n">col_offset</span><span class="p">,</span>
<span class="n">funcall_msg</span><span class="o">=</span><span class="n">funcall_msg</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">visit_Call</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">node</span><span class="p">):</span>
<span class="c1"># if we are in a translator function, issue a warninc</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">in_call</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">in_call</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">TRANSLATION_FUNCTIONS</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">"WARNING: function call within a translation function at "</span>
<span class="s2">"</span><span class="si">{filename}</span><span class="s2">:</span><span class="si">{line}</span><span class="s2">:</span><span class="si">{col}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">filename</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">filename</span><span class="p">,</span>
<span class="n">line</span><span class="o">=</span><span class="n">node</span><span class="o">.</span><span class="n">lineno</span><span class="p">,</span>
<span class="n">col</span><span class="o">=</span><span class="n">node</span><span class="o">.</span><span class="n">col_offset</span><span class="p">))</span>
<span class="n">funcname</span> <span class="o">=</span> <span class="n">get_func_name</span><span class="p">(</span><span class="n">node</span><span class="p">)</span>
<span class="k">for</span> <span class="n">arg</span> <span class="ow">in</span> <span class="n">node</span><span class="o">.</span><span class="n">args</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">visit_with_trace</span><span class="p">(</span><span class="n">arg</span><span class="p">,</span> <span class="n">funcname</span><span class="p">)</span>
<span class="k">for</span> <span class="n">kwarg</span> <span class="ow">in</span> <span class="n">node</span><span class="o">.</span><span class="n">keywords</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">visit_with_trace</span><span class="p">(</span><span class="n">kwarg</span><span class="o">.</span><span class="n">value</span><span class="p">,</span> <span class="n">funcname</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">generic_visit</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">node</span><span class="p">):</span>
<span class="c1"># if we are inside a translator function, issue a warning</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">in_call</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">in_call</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">TRANSLATION_FUNCTIONS</span><span class="p">:</span>
<span class="c1"># Some ast nodes, like Add don’t have position information</span>
<span class="k">if</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="s1">'lineno'</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">"WARNING: something not a string (</span><span class="si">{klass}</span><span class="s2">) found in a "</span>
<span class="s2">"translation function at </span><span class="si">{filename}</span><span class="s2">:</span><span class="si">{line}</span><span class="s2">:</span><span class="si">{col}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
<span class="n">filename</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">filename</span><span class="p">,</span>
<span class="n">klass</span><span class="o">=</span><span class="n">node</span><span class="o">.</span><span class="vm">__class__</span><span class="o">.</span><span class="vm">__name__</span><span class="p">,</span>
<span class="n">line</span><span class="o">=</span><span class="n">node</span><span class="o">.</span><span class="n">lineno</span><span class="p">,</span>
<span class="n">col</span><span class="o">=</span><span class="n">node</span><span class="o">.</span><span class="n">col_offset</span><span class="p">))</span>
<span class="k">else</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">"WARNING: something not a string (</span><span class="si">{klass}</span><span class="s2">) found in a "</span>
<span class="s2">"translation function. Position unknown; function call "</span>
<span class="s2">"is at </span><span class="si">{filename}</span><span class="s2">:</span><span class="si">{line}</span><span class="s2">:</span><span class="si">{col}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
<span class="n">filename</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">filename</span><span class="p">,</span>
<span class="n">klass</span><span class="o">=</span><span class="n">node</span><span class="o">.</span><span class="vm">__class__</span><span class="o">.</span><span class="vm">__name__</span><span class="p">,</span>
<span class="n">line</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">in_call</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">][</span><span class="mi">1</span><span class="p">],</span>
<span class="n">col</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">in_call</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">][</span><span class="mi">2</span><span class="p">]))</span>
<span class="nb">super</span><span class="p">(</span><span class="n">ShowStrings</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">generic_visit</span><span class="p">(</span><span class="n">node</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">tst</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="k">pass</span>
<span class="k">def</span> <span class="nf">actual_tests</span><span class="p">():</span>
<span class="n">_</span><span class="p">(</span><span class="s1">'translated 1'</span><span class="p">)</span>
<span class="n">tst</span><span class="p">(</span><span class="n">_</span><span class="p">(</span><span class="s1">'translated 2'</span><span class="p">))</span>
<span class="n">tst</span><span class="p">(</span><span class="n">gettext</span><span class="o">.</span><span class="n">gettext</span><span class="p">(</span><span class="s1">'translated 3'</span><span class="p">))</span>
<span class="n">tst</span><span class="p">(</span><span class="n">_</span><span class="p">(</span><span class="s1">'translated 4'</span><span class="p">)</span> <span class="o">+</span> <span class="s1">'native 1'</span><span class="p">)</span>
<span class="n">tst</span><span class="p">(</span><span class="s1">'native 2'</span>
<span class="s1">'native 3'</span><span class="p">)</span>
<span class="n">tst</span><span class="p">(</span><span class="n">_</span><span class="p">(</span><span class="s1">'native 4'</span> <span class="o">+</span> <span class="s1">'native 5'</span><span class="p">))</span>
<span class="n">tst</span><span class="p">(</span><span class="s1">'native 6'</span><span class="p">,</span> <span class="n">b</span><span class="o">=</span><span class="s1">'native 7'</span><span class="p">)</span>
<span class="n">tst</span><span class="p">(</span><span class="n">_</span><span class="p">(</span><span class="n">tst</span><span class="p">(</span><span class="s1">'hello!'</span><span class="p">)))</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">filename</span> <span class="o">=</span> <span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="k">except</span> <span class="ne">IndexError</span><span class="p">:</span>
<span class="n">filename</span> <span class="o">=</span> <span class="vm">__file__</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">"INFO: No filename specified, checking myself."</span><span class="p">)</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="s1">'r'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">code</span> <span class="o">=</span> <span class="n">f</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
<span class="n">root</span> <span class="o">=</span> <span class="n">ast</span><span class="o">.</span><span class="n">parse</span><span class="p">(</span><span class="n">code</span><span class="p">)</span>
<span class="n">show_strings</span> <span class="o">=</span> <span class="n">ShowStrings</span><span class="p">(</span><span class="n">filename</span><span class="o">=</span><span class="n">filename</span><span class="p">)</span>
<span class="n">show_strings</span><span class="o">.</span><span class="n">visit</span><span class="p">(</span><span class="n">root</span><span class="p">)</span>
</pre></div>
<p>The class initialization does two things: creates an empty <tt class="docutils literal">in_call</tt> list (this will hold our
primitive backtrace), and saves the filename, if provided.</p>
<p><tt class="docutils literal">visitCall</tt>, again, has two tasks. First, it checks if we are inside a translation function.
If so, it reports the fact that we are translating something that is not a raw string. Although
it is not necessarily a bad thing, I consider it bad practice as it may result in undefined behaviour.</p>
<p>Its second task is to walk through the positional and keyword arguments of the function call. For
each argument it calls the <tt class="docutils literal">visit_with_trace()</tt> method.</p>
<p>This method updates the <tt class="docutils literal">in_call</tt> property with the current function name and the position of
the call. This latter is needed because <tt class="docutils literal">ast</tt> doesn’t store position information for every node
(operators are a notable example). Then it simply visits the argument node, which is needed
because <tt class="docutils literal">NodeVisitor.visit()</tt> is not recursive. When the visit is done (which, with really
deeply nested calls like <tt class="docutils literal"><span class="pre">visit(this(call(iff(you(dare)))))</span></tt> will be recursive), the current
function name is removed from <tt class="docutils literal">in_call</tt>, so subsequent calls on the same level see the same “backtrace”.</p>
<p>The <tt class="docutils literal">generic_visit()</tt> method is called for every node that doesn’t have a named visitor (like
<tt class="docutils literal">visitCall</tt> or <tt class="docutils literal">visitStr</tt>. For the same reason we generate a warning in <tt class="docutils literal">visitCall</tt>, we do
the same here. If there is anything but a raw string inside a translation function call,
developers should know about it.</p>
<p>The last and I think the most important method is <tt class="docutils literal">visitStr</tt>. All it does is checking the last
element of the <tt class="docutils literal">in_call</tt> list, and generates a warning if a raw string is found somewhere that
is not inside a translation function call.</p>
<p>For accurate reports, there is a <tt class="docutils literal">get_func_name()</tt> function that takes an <tt class="docutils literal">ast</tt> node as an
argument. As function call can be anything from actual functions to object methods, this goes all
down the node’s properties, and recursively reconstructs the name of the actual function.</p>
<p>Finally, there are some test functions in this code. <tt class="docutils literal">tst</tt> and
<tt class="docutils literal">actual_tests</tt> are there so if I run a self-check on this script, it will
find these strings and report all the untranslated strings and all the
potential problems like the string concatenation.</p>
</div>
<div class="section" id="drawbacks">
<h2>Drawbacks</h2>
<p>There are several drawbacks here. First, translation function names are built in, to the
<tt class="docutils literal">TRANSLATION_FUNCTIONS</tt> property of the <tt class="docutils literal">ShowString</tt> class. You must change this if you use
other translation functions like <tt class="docutils literal">dngettext</tt>, or if you use a translation library other than
<tt class="docutils literal">gettext</tt>.</p>
<p>Second, it cannot ignore untranslated strings right now. It would be great if a pragma like
<tt class="docutils literal">flake8</tt>’s <tt class="docutils literal"># noqa</tt> or <tt class="docutils literal">coverage.py</tt>’s <tt class="docutils literal"># pragma: no cover</tt> could be added. However,
<tt class="docutils literal">ast</tt> doesn’t parse comment blocks, so this proves to be challenging.</p>
<p>Third, it reports docstrings as untranslated. Clearly, this is wrong, as docstrings generally
don’t have to be translated. Ignoring them, again, is a nice challenge I couldn’t yet overcome.</p>
<p>The <tt class="docutils literal">get_func_name()</tt> helper is everything but done. As long as I cannot remove that final
<tt class="docutils literal">else</tt> clause, there may be error reports. If that happens, the reported class should be
treated in a new <tt class="docutils literal">elif</tt> branch.</p>
<p>Finally (and the most easily fixed), the warnings are simply printed on the console. It is nice,
but it should be optional; the problems identified should be stored so the caller can obtain it as
an array.</p>
</div>
<div class="section" id="bottom-line">
<h2>Bottom line</h2>
<p>Finding strings in Python sources is not as hard as I imagined. It was fun to learn using the
<tt class="docutils literal">ast</tt> module, and it does a great job. Once I can overcome the drawbacks above, this script
will be a fantastic piece of code that can assist me in my future tasks.</p>
</div>
Slugify in Python 32016-12-08T12:54:19+00:002016-12-08T12:54:19+00:00Gergely Polonkaitag:None,2016-12-08:2016/12/08/slugify-in-python3/<p>Today I needed a function to create a slug (an <span class="caps">ASCII</span>-only representation of a string). I went
Googling a bit, and found an excellend <a class="reference external" href="http://flask.pocoo.org/snippets/5/">Flask snippet</a>.
Problem is, it is designed for Python 2, so I came up with a Python 3 version.</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">re</span>
<span class="kn">from</span> <span class="nn">unicodedata</span> <span class="kn">import</span> <span class="n">normalize …</span></pre></div><p>Today I needed a function to create a slug (an <span class="caps">ASCII</span>-only representation of a string). I went
Googling a bit, and found an excellend <a class="reference external" href="http://flask.pocoo.org/snippets/5/">Flask snippet</a>.
Problem is, it is designed for Python 2, so I came up with a Python 3 version.</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">re</span>
<span class="kn">from</span> <span class="nn">unicodedata</span> <span class="kn">import</span> <span class="n">normalize</span>
<span class="n">_punctuation_re</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="sa">r</span><span class="s1">'[\t !"#$%&</span><span class="se">\'</span><span class="s1">()*\-/<=>?@\[</span><span class="se">\\</span><span class="s1">\]^_`{|},.]+'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">slugify</span><span class="p">(</span><span class="n">text</span><span class="p">,</span> <span class="n">delim</span><span class="o">=</span><span class="s1">'-'</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""</span>
<span class="sd"> Generate an ASCII-only slug.</span>
<span class="sd"> """</span>
<span class="n">result</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">word</span> <span class="ow">in</span> <span class="n">_punctuation_re</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">text</span><span class="o">.</span><span class="n">lower</span><span class="p">()):</span>
<span class="n">word</span> <span class="o">=</span> <span class="n">normalize</span><span class="p">(</span><span class="s1">'NFKD'</span><span class="p">,</span> <span class="n">word</span><span class="p">)</span> \
<span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">'ascii'</span><span class="p">,</span> <span class="s1">'ignore'</span><span class="p">)</span> \
<span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">'utf-8'</span><span class="p">)</span>
<span class="k">if</span> <span class="n">word</span><span class="p">:</span>
<span class="n">result</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">word</span><span class="p">)</span>
<span class="k">return</span> <span class="n">delim</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">result</span><span class="p">)</span>
</pre></div>
<p>As I don’t really like the transliteration done in the first example (e.g. converting ü to ue), I
went with the second example.</p>
Add Python docstring to the beginning of anything in Emacs2016-11-30T07:52:37+00:002016-11-30T07:52:37+00:00Gergely Polonkaitag:None,2016-11-30:2016/11/30/add-python-docstring-to-the-beginning-of-anything/<p>Now that I write Python code for a living, I write a lot of functions, classes, and modules. What
I still tend to forget, and also find tedious, is adding docstrings. Unlike many developers,
writing documentation is not an enemy of mine, but it usually comes to my mind when …</p><p>Now that I write Python code for a living, I write a lot of functions, classes, and modules. What
I still tend to forget, and also find tedious, is adding docstrings. Unlike many developers,
writing documentation is not an enemy of mine, but it usually comes to my mind when I finish
implementation. The procedure, roughly, is this:</p>
<ul class="simple">
<li>Decide where I am (in a function, in a class but not in one of its methods, or not inside such a
block at all)</li>
<li>Go to the beginning of the thing</li>
<li>Insert <tt class="docutils literal">"""</tt></li>
<li>Leave a blank line</li>
<li>Insert <tt class="docutils literal">"""</tt></li>
</ul>
<p>One of my mottos is if something takes more than one step and you have to do it more than twice,
you should automate it after the first time. This puts a small(ish) overhead on the second
invocation (when you implement the automation), but it usually worth the time.</p>
<p>Since I use Emacs for pretty much everything coding-related (and many more, but that’s the topic
of a different post), I wrote a small function to do it for me.</p>
<div class="highlight"><pre><span></span><span class="p">(</span><span class="nb">defun</span><span class="w"> </span><span class="nv">gpolonkai/prog-in-string-p</span><span class="w"> </span><span class="p">()</span>
<span class="w"> </span><span class="s">"Return `t' if point is inside a string."</span>
<span class="w"> </span><span class="p">(</span><span class="nb">nth</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="p">(</span><span class="nv">syntax-ppss</span><span class="p">)))</span>
<span class="p">(</span><span class="nb">defun</span><span class="w"> </span><span class="nv">gpolonkai/prog-in-comment-p</span><span class="w"> </span><span class="p">()</span>
<span class="w"> </span><span class="s">"Return `t' if point is inside a comment."</span>
<span class="w"> </span><span class="p">(</span><span class="nb">nth</span><span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="p">(</span><span class="nv">syntax-ppss</span><span class="p">)))</span>
<span class="p">(</span><span class="nb">defun</span><span class="w"> </span><span class="nv">gpolonkai/python-add-docstring</span><span class="w"> </span><span class="p">()</span>
<span class="w"> </span><span class="s">"Add a Python docstring to the current thing. If point is</span>
<span class="s"> inside a function, add docstring to that. If point is in a</span>
<span class="s"> class, add docstring to that. If neither, add docstring to the</span>
<span class="s"> beginning of the file."</span>
<span class="w"> </span><span class="p">(</span><span class="nv">interactive</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nv">save-restriction</span>
<span class="w"> </span><span class="p">(</span><span class="nv">widen</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nv">beginning-of-defun</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nb">not</span><span class="w"> </span><span class="p">(</span><span class="nv">looking-at-p</span><span class="w"> </span><span class="s">"\\(def\\|class\\) "</span><span class="p">))</span>
<span class="w"> </span><span class="p">(</span><span class="k">progn</span>
<span class="w"> </span><span class="p">(</span><span class="nv">goto-char</span><span class="w"> </span><span class="p">(</span><span class="nv">point-min</span><span class="p">))</span>
<span class="w"> </span><span class="p">(</span><span class="nv">back-to-indentation</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nv">forward-char</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nv">while</span><span class="w"> </span><span class="p">(</span><span class="nv">gpolonkai/prog-in-comment-p</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nv">forward-line</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nv">back-to-indentation</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nv">forward-char</span><span class="p">)))</span>
<span class="w"> </span><span class="p">(</span><span class="nv">search-forward</span><span class="w"> </span><span class="s">":"</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nv">while</span><span class="w"> </span><span class="p">(</span><span class="nb">or</span><span class="w"> </span><span class="p">(</span><span class="nv">gpolonkai/prog-in-string-p</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nv">gpolonkai/prog-in-comment-p</span><span class="p">))</span>
<span class="w"> </span><span class="p">(</span><span class="nv">search-forward</span><span class="w"> </span><span class="s">":"</span><span class="p">)))</span>
<span class="w"> </span><span class="p">(</span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nb">eq</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">(</span><span class="nv">count-lines</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">(</span><span class="nv">point</span><span class="p">)))</span>
<span class="w"> </span><span class="p">(</span><span class="nv">open-line-above</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nv">open-line-below</span><span class="p">))</span>
<span class="w"> </span><span class="p">(</span><span class="nv">insert</span><span class="w"> </span><span class="s">"\"\"\""</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nv">open-line-below</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nv">insert</span><span class="w"> </span><span class="s">"\"\"\""</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nv">open-line-above</span><span class="p">)))</span>
</pre></div>
<p>There are still a lot of things to improve:</p>
<ul class="simple">
<li>it always inserts double quotes (althoug I couldn’t show a use-case when single quotes are preferred)</li>
<li>it doesn’t check for an existing docstring, just happily inserts a new one (leaving the old one
alone, but generating a syntax error this way)</li>
<li>it would also be nice if I could jump to the beginning of a file even from a class method. I
guess I will use prefix keys for that, but I’m not sure yet.</li>
</ul>
<p>You can bet I will implement these features, so check back soon for an updated version!</p>
Get account data programatically from id-manager2016-11-18T12:43:13+00:002016-11-18T12:43:13+00:00Gergely Polonkaitag:None,2016-11-18:2016/11/18/get-passwords-from-id-manager/<p>I recently started using <a class="reference external" href="https://github.com/kiwanami/emacs-id-manager">id-manager</a>. It is a
nice little package that can store your passwords, encrypting them with <span class="caps">GPG</span>. My original reason
was to store my GitHub access token for <a class="reference external" href="https://github.com/xuchunyang/github-notifier.el">github-notifier</a>, but it soon turned out, it’s not <em>that</em> easy.</p>
<p><tt class="docutils literal"><span class="pre">id-manager</span></tt> is a nice package when it comes …</p><p>I recently started using <a class="reference external" href="https://github.com/kiwanami/emacs-id-manager">id-manager</a>. It is a
nice little package that can store your passwords, encrypting them with <span class="caps">GPG</span>. My original reason
was to store my GitHub access token for <a class="reference external" href="https://github.com/xuchunyang/github-notifier.el">github-notifier</a>, but it soon turned out, it’s not <em>that</em> easy.</p>
<p><tt class="docutils literal"><span class="pre">id-manager</span></tt> is a nice package when it comes to storing your password and retrieving them for
your own eyes. But it cannot retrieve account data programatically. Taking a look into its
source code, I came up with this solution:</p>
<div class="highlight"><pre><span></span><span class="p">(</span><span class="nb">defun</span><span class="w"> </span><span class="nv">gpolonkai/idm-record-get-field</span><span class="w"> </span><span class="p">(</span><span class="nv">record</span><span class="w"> </span><span class="nv">field</span><span class="p">)</span>
<span class="w"> </span><span class="s">"Get FIELD of an id-manager RECORD."</span>
<span class="w"> </span><span class="p">(</span><span class="k">let</span><span class="w"> </span><span class="p">((</span><span class="nv">funcname</span><span class="w"> </span><span class="p">(</span><span class="nb">intern</span><span class="w"> </span><span class="p">(</span><span class="nv">concat</span><span class="w"> </span><span class="s">"idm-record-"</span><span class="w"> </span><span class="p">(</span><span class="nb">symbol-name</span><span class="w"> </span><span class="nv">field</span><span class="p">)))))</span>
<span class="w"> </span><span class="p">(</span><span class="nb">when</span><span class="w"> </span><span class="p">(</span><span class="nb">fboundp</span><span class="w"> </span><span class="nv">funcname</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nb">funcall</span><span class="w"> </span><span class="nv">funcname</span><span class="w"> </span><span class="nv">record</span><span class="p">))))</span>
<span class="p">(</span><span class="nb">defun</span><span class="w"> </span><span class="nv">gpolonkai/idm-get-field-for-account</span><span class="w"> </span><span class="p">(</span><span class="nv">account</span><span class="w"> </span><span class="nv">field</span><span class="p">)</span>
<span class="w"> </span><span class="s">"Get id-manager password for ACCOUNT."</span>
<span class="w"> </span><span class="p">(</span><span class="k">let</span><span class="w"> </span><span class="p">((</span><span class="nv">db</span><span class="w"> </span><span class="p">(</span><span class="nv">idm-load-db</span><span class="p">))</span>
<span class="w"> </span><span class="p">(</span><span class="nv">lookup-record</span><span class="w"> </span><span class="no">nil</span><span class="p">))</span>
<span class="w"> </span><span class="p">(</span><span class="nb">dolist</span><span class="w"> </span><span class="p">(</span><span class="nv">record</span><span class="w"> </span><span class="p">(</span><span class="nb">funcall</span><span class="w"> </span><span class="nv">db</span><span class="w"> </span><span class="ss">'get-all-records</span><span class="p">)</span><span class="w"> </span><span class="nv">password</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nb">when</span><span class="w"> </span><span class="p">(</span><span class="nb">string=</span><span class="w"> </span><span class="nv">account</span><span class="w"> </span><span class="p">(</span><span class="nv">idm-record-name</span><span class="w"> </span><span class="nv">record</span><span class="p">))</span>
<span class="w"> </span><span class="p">(</span><span class="k">setq</span><span class="w"> </span><span class="nv">lookup-record</span><span class="w"> </span><span class="p">(</span><span class="nv">gpolonkai/idm-record-get-field</span><span class="w"> </span><span class="nv">record</span><span class="w"> </span><span class="nv">field</span><span class="p">))))</span>
<span class="w"> </span><span class="nv">lookup-record</span><span class="p">))</span>
<span class="p">(</span><span class="nb">defmacro</span><span class="w"> </span><span class="nv">gpolonkai/idm-get-password-for-account</span><span class="w"> </span><span class="p">(</span><span class="nv">account</span><span class="p">)</span>
<span class="w"> </span><span class="o">`</span><span class="p">(</span><span class="nv">gpolonkai/idm-get-field-for-account</span><span class="w"> </span><span class="o">,</span><span class="nv">account</span><span class="w"> </span><span class="ss">'password</span><span class="p">))</span>
<span class="p">(</span><span class="nb">defmacro</span><span class="w"> </span><span class="nv">gpolonkai/idm-get-id-for-account</span><span class="w"> </span><span class="p">(</span><span class="nv">account</span><span class="p">)</span>
<span class="w"> </span><span class="o">`</span><span class="p">(</span><span class="nv">gpolonkai/idm-get-field-for-account</span><span class="w"> </span><span class="o">,</span><span class="nv">account</span><span class="w"> </span><span class="ss">'account-id</span><span class="p">))</span>
</pre></div>
<p>I currently need only the account <span class="caps">ID</span> (ie. the username) and the password, but it’s pretty easy to
add a macro to get the <tt class="docutils literal">memo</tt> or <tt class="docutils literal"><span class="pre">update-time</span></tt> fields, too.</p>
Edit file as another user in Emacs2016-11-10T08:57:12+00:002016-11-10T08:57:12+00:00Gergely Polonkaitag:None,2016-11-10:2016/11/10/edit-file-as-other-user-in-emacs/<p>I have recently found <a class="reference external" href="http://emacsredux.com/blog/2013/04/21/edit-files-as-root/">this article</a>
by Bozhidar Batsov on opening the current file as root. I barely use <a class="reference external" href="https://www.gnu.org/software/tramp/">tramp</a> for sudo access, but when I do, I almost never use root as
the target user. So I decided to fix it for my needs.</p>
<div class="highlight"><pre><span></span><span class="p">(</span><span class="nb">defun</span><span class="w"> </span><span class="nv">open-this-file-as-other-user</span><span class="w"> </span><span class="p">(</span><span class="nv">user</span><span class="p">)</span>
<span class="w"> </span><span class="s">"Edit current …</span></pre></div><p>I have recently found <a class="reference external" href="http://emacsredux.com/blog/2013/04/21/edit-files-as-root/">this article</a>
by Bozhidar Batsov on opening the current file as root. I barely use <a class="reference external" href="https://www.gnu.org/software/tramp/">tramp</a> for sudo access, but when I do, I almost never use root as
the target user. So I decided to fix it for my needs.</p>
<div class="highlight"><pre><span></span><span class="p">(</span><span class="nb">defun</span><span class="w"> </span><span class="nv">open-this-file-as-other-user</span><span class="w"> </span><span class="p">(</span><span class="nv">user</span><span class="p">)</span>
<span class="w"> </span><span class="s">"Edit current file as USER, using `tramp' and `sudo'. If the current</span>
<span class="s">buffer is not visiting a file, prompt for a file name."</span>
<span class="w"> </span><span class="p">(</span><span class="nv">interactive</span><span class="w"> </span><span class="s">"sEdit as user (default: root): "</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nb">when</span><span class="w"> </span><span class="p">(</span><span class="nb">string=</span><span class="w"> </span><span class="s">""</span><span class="w"> </span><span class="nv">user</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="k">setq</span><span class="w"> </span><span class="nv">user</span><span class="w"> </span><span class="s">"root"</span><span class="p">))</span>
<span class="w"> </span><span class="p">(</span><span class="k">let*</span><span class="w"> </span><span class="p">((</span><span class="nv">filename</span><span class="w"> </span><span class="p">(</span><span class="nb">or</span><span class="w"> </span><span class="nv">buffer-file-name</span>
<span class="w"> </span><span class="p">(</span><span class="nv">read-file-name</span><span class="w"> </span><span class="p">(</span><span class="nb">format</span><span class="w"> </span><span class="s">"Find file (as %s): "</span>
<span class="w"> </span><span class="nv">user</span><span class="p">))))</span>
<span class="w"> </span><span class="p">(</span><span class="nv">tramp-path</span><span class="w"> </span><span class="p">(</span><span class="nv">concat</span><span class="w"> </span><span class="p">(</span><span class="nb">format</span><span class="w"> </span><span class="s">"/sudo:%s@localhost:"</span><span class="w"> </span><span class="nv">user</span><span class="p">)</span><span class="w"> </span><span class="nv">filename</span><span class="p">)))</span>
<span class="w"> </span><span class="p">(</span><span class="k">if</span><span class="w"> </span><span class="nv">buffer-file-name</span>
<span class="w"> </span><span class="p">(</span><span class="nv">find-alternate-file</span><span class="w"> </span><span class="nv">tramp-path</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nv">find-file</span><span class="w"> </span><span class="nv">tramp-path</span><span class="p">))))</span>
</pre></div>
<p>If the user is not specified, the default is still root. Also, if the current buffer is not
visiting a file, I prompt for a filename. As I’m not an <tt class="docutils literal">ido</tt> user, I didn’t bother calling
<tt class="docutils literal"><span class="pre">ido-read-file-name</span></tt>; <a class="reference external" href="https://github.com/emacs-helm/helm/wiki">helm</a> overrides
<tt class="docutils literal"><span class="pre">read-file-name</span></tt> for me anyway.</p>
<p>Unlike Bozhidar, I barely use this feature, so I didn’t bind this to a key.</p>
How I started with Emacs2016-11-03T09:58:41+00:002016-11-03T09:58:41+00:00Gergely Polonkaitag:None,2016-11-03:2016/11/03/how-i-started-with-emacs/<p>Sacha Chua has a nice <a class="reference external" href="http://sachachua.com/blog/2013/04/emacs-chat-intro/">Emacs chat intro</a>
article back from 2013. I write this post half because she asks there about my (<span class="caps">OK</span>, anyone’s)
first Emacs moments, and half because I plan to do it for months now.</p>
<p>I wanted to start using Emacs 6(ish) years ago …</p><p>Sacha Chua has a nice <a class="reference external" href="http://sachachua.com/blog/2013/04/emacs-chat-intro/">Emacs chat intro</a>
article back from 2013. I write this post half because she asks there about my (<span class="caps">OK</span>, anyone’s)
first Emacs moments, and half because I plan to do it for months now.</p>
<p>I wanted to start using Emacs 6(ish) years ago, and I was like “<kbd>C-x</kbd> what”? (Note that back
around 1998, I was among the people who exited <tt class="docutils literal">vi</tt> by killing it from another terminal after a
bunch of tries <span class="amp">&</span> fails like <a class="reference external" href="http://osxdaily.com/2014/06/12/how-to-quit-vim/">these</a>.)</p>
<p>I tried to come back to Emacs a lot of times. And I mean a <em>lot</em>, about every two months. I
suddenly learned what these cryptic key chord descriptions mean (<tt class="docutils literal">C</tt> is for <kbd>Control</kbd> and
<tt class="docutils literal">M</tt> is for <kbd>Meta</kbd>, which is actually <kbd>Alt</kbd>), but somehow it didn’t <em>click</em>. I
remained a ViM power user with a huge pile of 3:sup:<cite>rd</cite> party plugins. Then <a class="reference external" href="blog/2014/9/17/nyanmacs.html">I found Nyan-macs</a>), which converted me to Emacs, and it is final now. Many of
my friends thought I’m just kidding this being the cause, but I’m not. I’m a huge fan of Nyan cat
(did you know there is even a site called <a class="reference external" href="http://nyan.cat/">nyan.cat</a>?) and since then I have
it in my mode line:</p>
<img alt="Nyan modeline" src="images/nyan-modeline.png" />
<p>…in my <tt class="docutils literal">eshell</tt> prompt:</p>
<img alt="eshell prompt with a Nyan cat" src="images/nyan-eshell.png" />
<p>…and I also <a class="reference external" href="https://www.emacswiki.org/emacs/ZoneMode">zone out</a> with Nyan cat:</p>
<img alt="a text-based animation with Nyan cat" src="images/nyan-zone.png" />
<p>Now on to more serious stuff. After browsing through all the packages provided by <a class="reference external" href="http://elpa.gnu.org/"><span class="caps">ELPA</span></a>, I found tons of useful (and sometimes, less useful) packages, like <a class="reference external" href="https://github.com/emacs-helm/helm/wiki">Helm</a>, <a class="reference external" href="http://company-mode.github.io/">company</a>, <a class="reference external" href="https://www.emacswiki.org/emacs/GnuGlobal">gtags</a> (which introduced me to <span class="caps">GNU</span> Global, removing
Exuberant ctags from my life), <a class="reference external" href="https://magit.vc/">magit</a>, <a class="reference external" href="http://batsov.com/projectile/">Projectile</a>, and <a class="reference external" href="http://orgmode.org/">Org</a> (<span class="caps">OK</span>, it’s actually part of
Emacs for a while, but still). I still use these few, but in a month or two, I started to
<a class="reference external" href="https://github.com/gergelypolonkai/my-emacs-d">version control</a> my <tt class="docutils literal">.emacs.d</tt> directory, so I
can easily transfer it between my home and work machine (and for a few weeks now, even to my
phone: I’m using Termux on Android). Then, over these two years I wrote some packages like
<a class="reference external" href="https://github.com/gergelypolonkai/gobgen.el">GobGen</a>, and a small addon for Calendar providing
<a class="reference external" href="https://github.com/gergelypolonkai/hungarian-holidays">Hungarian holidays</a>, and I found a lot
more (in no particular order):</p>
<ul class="simple">
<li><a class="reference external" href="https://github.com/syohex/emacs-git-gutter">git-gutter</a></li>
<li><a class="reference external" href="https://github.com/magnars/multiple-cursors.el">multiple-cursors</a></li>
<li><a class="reference external" href="https://github.com/gregsexton/origami.el">origami</a></li>
<li><a class="reference external" href="https://github.com/abo-abo/ace-window">ace-window</a></li>
<li><a class="reference external" href="https://github.com/abo-abo/avy">avy</a></li>
<li><a class="reference external" href="https://github.com/Malabarba/beacon">beacon</a></li>
</ul>
<p>…and a lot more.</p>
<p>What is more important (to me) is that I started using the <a class="reference external" href="https://github.com/jwiegley/use-package">use-package</a> package, which can automatically download packages
that are not installed on my current local system. Together with <a class="reference external" href="https://github.com/rranelli/auto-package-update.el">auto-package-update</a>, it is <em>very</em> practical.</p>
<p>In addition, I started to follow the blogs of a bunch of Emacs users/gurus. I’ve already
mentioned <a class="reference external" href="http://sachachua.com/">Sacha Chua</a>. She’s a charming, cheerful person, writing a lot
about Emacs and project management (among other things). Another one is <a class="reference external" href="http://batsov.com/">Bozhidar Batsov</a>, who, among other things, had an initiate to lay down the foundation of a
<a class="reference external" href="https://github.com/bbatsov/emacs-lisp-style-guide">common Elisp coding style</a>. Another
favourite of mine is <a class="reference external" href="http://endlessparentheses.com/">Endless Parentheses</a>, whence I got a lot
of ideas.</p>
git-merge stages2016-10-04T12:46:00+00:002016-10-04T12:46:00+00:00Gergely Polonkaitag:None,2016-10-04:2016/10/04/git-merge-stages/<p>This was a mail to my company’s internal Git mailing list, after I realised many colleagues can’t
wrap their heads around merge conflicts.</p>
<blockquote>
<p>Hello all,</p>
<p>I just saw this on the <a class="reference external" href="https://groups.google.com/forum/#!forum/git-users">git-users</a> list
and thought it could help you when you bump into a merge conflict. It is …</p></blockquote><p>This was a mail to my company’s internal Git mailing list, after I realised many colleagues can’t
wrap their heads around merge conflicts.</p>
<blockquote>
<p>Hello all,</p>
<p>I just saw this on the <a class="reference external" href="https://groups.google.com/forum/#!forum/git-users">git-users</a> list
and thought it could help you when you bump into a merge conflict. It is an excerpt from a
mail by Konstantin Khomoutov (one of the main contributors on the list), with a few
modifications of mine. Happy debugging :)</p>
<p>When a merge conflict is detected for a file, Git:</p>
<ol class="arabic simple">
<li>Updates the entry for that file in the index to make it contain
several so-called “stages”:<ul>
<li><cite>0</cite>: “Ours” version – that one which was there in this index entry
before we begun to merge. At the beginning of the conflict, like
right after the <cite>git merge</cite> or <cite>git rebase</cite> command this won’t
exist (unless you had the file in the index, which you didn’t, did
you?). When you resolve the conflict and use <cite>git add
my/conflicting/file.cc</cite>, this will be the version added to the
staging area (index), thus, the resolution of the conflict.</li>
<li><cite>1</cite>: The version from the common ancestor commit, ie. the version
of the file both of you modified.</li>
<li><cite>2</cite>: The version from <cite><span class="caps">HEAD</span></cite>. During a merge, this is the current
branch. During a rebase, this is the branch or commit you are
rebasing onto, which usually will be <cite>origin/develop</cite>).</li>
<li><cite>3</cite>: The version being merged, or the commit you are rebasing.</li>
</ul>
</li>
<li>Updates the file in the work tree to contain conflict markers and
the conflicting chunks of text between them (and the text from the
common ancestor if the <cite>diff3</cite> style of conflict markers was set).</li>
</ol>
<p>Now you can use the numbers in point 1 to access the different stages
of the conflicting file. For example, to see the common ancestor (the
version both of you modified), use</p>
<div class="highlight"><pre><span></span>git<span class="w"> </span>show<span class="w"> </span>:1:my/conflicting/file.cc
</pre></div>
<p>Or, to see the difference between the two conflicting versions, try</p>
<div class="highlight"><pre><span></span>git<span class="w"> </span>diff<span class="w"> </span>:2:my/conflicting/file.cc<span class="w"> </span>:3:my/conflicting/file.cc
</pre></div>
<p><strong>Note</strong> that you can’t use the <tt class="docutils literal">:0:</tt> stage <em>before</em> you stage your resolution with <tt class="docutils literal">git
add</tt>, and you can’t use the <tt class="docutils literal">:2:</tt> and <tt class="docutils literal">:3:</tt> stages <em>after</em> you staged the resolution.</p>
<p>Fun fact: behind the scenes, these are the files (<em>revisions</em>) <tt class="docutils literal">git mergetool</tt> accesses when
it presents you the conflict visually.</p>
</blockquote>
Emacs package to generate GObject boilerplate2016-09-28T15:40:15+00:002016-09-28T15:40:15+00:00Gergely Polonkaitag:None,2016-09-28:2016/09/28/emacs-package-to-generate-gobject-boilerplate/<p>Before I started using Vala (and sometimes even after that) I often needed to generate new classes
based on <a class="reference external" href="https://developer.gnome.org/gobject/stable/">GObject</a>.</p>
<p>If you have ever worked with GObject in C, you know how tedious it can be. You need a pretty long
boilerplate just to register your class, and, if you …</p><p>Before I started using Vala (and sometimes even after that) I often needed to generate new classes
based on <a class="reference external" href="https://developer.gnome.org/gobject/stable/">GObject</a>.</p>
<p>If you have ever worked with GObject in C, you know how tedious it can be. You need a pretty long
boilerplate just to register your class, and, if you want to be introspectable (and readable,
actually), your function names can grow really long.</p>
<p>To overcome this problem back in my ViM days, I used template files, where I could replace class
prefixes and names with a few keyboard macros. As I never really dug into ViM scripting other
than using some plugins, I never got farther than that. <a class="reference external" href="blog/2014/9/17/nyanmacs.html">Then came Emacs</a>.</p>
<p>I use Emacs for about two years now very extensively, up to and including GLib-based development.
I tried the template approach, but it felt to be a really poor experience, especially given that I
made my feet wet with Emacs Lisp. So I dug deeper, and created a package for that.</p>
<img alt="A screenshot of GobGen in action" src="images/screenshot-gobgen.png" />
<p>GobGen has its own buffer with some widgets, a bit similar to <tt class="docutils literal">customize</tt>. You can enter the
name of your new object and its parent, specify some settings. Then you press Generate, and you
are presented with two new buffers, one for the <tt class="docutils literal">.c</tt> and another for the <tt class="docutils literal">.h</tt> boilerplate.</p>
<p>There are a lot of things to do, actually. There is already an open issue for creating a major
mode for this buffer, and there are some minor switches I’d like to add, but it is already usable.
You can grab it from <a class="reference external" href="https://melpa.org/#/gobgen"><span class="caps">MELPA</span></a> (my first package there; woo!) or from
my <a class="reference external" href="https://github.com/gergelypolonkai/gobgen.el">GitHub account</a>.</p>
Vala interface madness2016-02-26T13:07:52+00:002016-02-26T13:07:52+00:00"Gergely Polonkai"tag:None,2016-02-26:2016/02/26/vala-interface-madness/<p>Although I have just started making it in C, I decided to move my Matrix GLib <span class="caps">SDK</span> to Vala. First
to learn a new language, and second because it is much easier to write GObject based stuff with it.</p>
<p>For the first step I created a <tt class="docutils literal">.vapi</tt> file from my …</p><p>Although I have just started making it in C, I decided to move my Matrix GLib <span class="caps">SDK</span> to Vala. First
to learn a new language, and second because it is much easier to write GObject based stuff with it.</p>
<p>For the first step I created a <tt class="docutils literal">.vapi</tt> file from my existing sources, so the whole <span class="caps">SDK</span> prototype
was available for me in Vala.</p>
<p>I had a <tt class="docutils literal">MatrixEvent</tt> class that implemented the <tt class="docutils literal">GInitable</tt> interface, and many others were
subclassed <tt class="docutils literal">MatrixEvent</tt>. For some reason I don’t remember, I created the following header for
one of the event classes:</p>
<div class="highlight"><pre><span></span><span class="kd">public</span><span class="w"> </span><span class="kd">class</span><span class="w"> </span><span class="nc">MatrixPresenceEvent</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="n">GLib</span><span class="p">.</span><span class="n">Object</span><span class="p">,</span><span class="w"> </span><span class="n">GLib</span><span class="p">.</span><span class="n">Initable</span><span class="w"> </span><span class="p">{</span>
</pre></div>
<p>This is nice and everything, but as I didn’t create an <tt class="docutils literal">init()</tt> method for
<tt class="docutils literal">MatrixPresenceEvent</tt>, it tried to use the one from the parent class and somehow got into an
infinite loop. The Vala transformer (<tt class="docutils literal">valac</tt>), however, doesn’t mention this.</p>
<p>Lessons learned: if you implement an interface on a subclass that is implemented by the parent
don’t forget to add the necessary functions to the subclass.</p>
Emacs: Implement a GObject’s virtual function2016-01-13T13:31:12+00:002016-01-13T13:31:12+00:00"Gergely Polonkai"tag:None,2016-01-13:2016/01/13/emacs-implement-a-gobject-s-virtual-function/<p>I have recently started creating a GLib implementation of the Matrix.org <span class="caps">API</span>. For that, I have
created a GObject interface, MatrixAPI, which has as many virtual functions as <span class="caps">API</span> calls (which is
a lot, and expanding). This way I ended up with the following scenario.</p>
<p>In <tt class="docutils literal"><span class="pre">matrix-api.h</span></tt> I …</p><p>I have recently started creating a GLib implementation of the Matrix.org <span class="caps">API</span>. For that, I have
created a GObject interface, MatrixAPI, which has as many virtual functions as <span class="caps">API</span> calls (which is
a lot, and expanding). This way I ended up with the following scenario.</p>
<p>In <tt class="docutils literal"><span class="pre">matrix-api.h</span></tt> I had a struct like this, with a lot more elements:</p>
<div class="highlight"><pre><span></span><span class="k">typedef</span><span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kt">void</span><span class="w"> </span><span class="p">(</span><span class="o">*</span><span class="n">initial_sync</span><span class="p">)(</span><span class="n">MatrixAPI</span><span class="w"> </span><span class="o">*</span><span class="n">api</span><span class="p">,</span>
<span class="w"> </span><span class="n">MatrixAPICallback</span><span class="w"> </span><span class="n">callback</span><span class="p">,</span>
<span class="w"> </span><span class="n">gpointer</span><span class="w"> </span><span class="n">user_data</span><span class="p">,</span>
<span class="w"> </span><span class="n">GError</span><span class="w"> </span><span class="o">**</span><span class="n">error</span><span class="p">);</span>
<span class="w"> </span><span class="kt">void</span><span class="w"> </span><span class="p">(</span><span class="o">*</span><span class="n">sync</span><span class="p">)(</span><span class="n">MatrixAPI</span><span class="w"> </span><span class="o">*</span><span class="n">api</span><span class="p">,</span>
<span class="w"> </span><span class="n">MatrixAPICallback</span><span class="w"> </span><span class="n">callback</span><span class="p">,</span>
<span class="w"> </span><span class="n">gpointer</span><span class="w"> </span><span class="n">user_data</span><span class="p">,</span>
<span class="w"> </span><span class="n">GError</span><span class="w"> </span><span class="o">**</span><span class="n">error</span><span class="p">);</span>
<span class="w"> </span><span class="err">…</span>
</pre></div>
<p>And in <tt class="docutils literal"><span class="pre">matrix-http-api.c</span></tt>, which implements <tt class="docutils literal">MatrixAPI</tt>, I have a function like this (again,
with a lot more elements):</p>
<div class="highlight"><pre><span></span><span class="k">static</span><span class="w"> </span><span class="kt">void</span>
<span class="nf">matrix_http_api_matrix_api_init</span><span class="p">(</span><span class="n">GObjectInterface</span><span class="w"> </span><span class="o">*</span><span class="n">iface</span><span class="p">)</span>
<span class="p">{</span>
<span class="w"> </span><span class="n">iface</span><span class="o">-></span><span class="n">initial_sync</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">i_initial_sync</span><span class="p">;</span>
<span class="w"> </span><span class="n">iface</span><span class="o">-></span><span class="n">sync</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">i_sync</span><span class="p">;</span>
<span class="w"> </span><span class="err">…</span>
<span class="p">}</span>
</pre></div>
<p>And every time I wanted to implement a new function from the vtable, I had to copy the prototype,
and add an <tt class="docutils literal"><span class="pre">iface->foo_bar</span> = i_foo_bar</tt> line and an actual function header for <tt class="docutils literal">i_foo_bar</tt>
with the same parameters. That’s a cumbersome job for more than 40 function headers. But Emacs
comes to the rescue!</p>
<div class="highlight"><pre><span></span><span class="p">(</span><span class="nb">require</span><span class="w"> </span><span class="ss">'thingatpt</span><span class="p">)</span>
<span class="p">(</span><span class="nb">defun</span><span class="w"> </span><span class="nv">get-point</span><span class="p">(</span><span class="nc">symbol</span><span class="w"> </span><span class="k">&optional</span><span class="w"> </span><span class="nv">arg</span><span class="p">)</span>
<span class="w"> </span><span class="s">"Get point, optionally running a command beforehand"</span>
<span class="w"> </span><span class="p">(</span><span class="nb">funcall</span><span class="w"> </span><span class="nc">symbol</span><span class="w"> </span><span class="nv">arg</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nv">point</span><span class="p">))</span>
<span class="p">(</span><span class="nb">defun</span><span class="w"> </span><span class="nv">copy-symbol-at-point</span><span class="p">()</span>
<span class="w"> </span><span class="s">"Copy the symbol under point"</span>
<span class="w"> </span><span class="p">(</span><span class="nv">interactive</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nv">save-excursion</span>
<span class="w"> </span><span class="p">(</span><span class="k">let</span><span class="w"> </span><span class="p">((</span><span class="nv">beg</span><span class="w"> </span><span class="p">(</span><span class="nv">get-point</span><span class="w"> </span><span class="ss">'beginning-of-thing</span><span class="w"> </span><span class="ss">'symbol</span><span class="p">))</span>
<span class="w"> </span><span class="p">(</span><span class="nv">end</span><span class="w"> </span><span class="p">(</span><span class="nv">get-point</span><span class="w"> </span><span class="ss">'end-of-thing</span><span class="w"> </span><span class="ss">'symbol</span><span class="p">)))</span>
<span class="w"> </span><span class="p">(</span><span class="nv">copy-region-as-kill</span><span class="w"> </span><span class="nv">beg</span><span class="w"> </span><span class="nv">end</span><span class="p">))))</span>
<span class="p">(</span><span class="nb">defun</span><span class="w"> </span><span class="nv">implement-gobject-vfunc</span><span class="p">()</span>
<span class="w"> </span><span class="s">"Change a vtable line of a GObject interface to an implementation line like:</span>
<span class="s">void (*my_iface_func)(type1 param1, type2 param2, ...);</span>
<span class="s">to</span>
<span class="s">iface->my_iface_func = i_my_iface_func;"</span>
<span class="w"> </span><span class="p">(</span><span class="nv">interactive</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nv">save-excursion</span>
<span class="w"> </span><span class="p">(</span><span class="k">let</span><span class="w"> </span><span class="p">((</span><span class="nv">beg</span><span class="w"> </span><span class="p">((</span><span class="k">lambda</span><span class="p">()</span>
<span class="w"> </span><span class="p">(</span><span class="nv">search-forward</span><span class="w"> </span><span class="s">"(*"</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nv">point</span><span class="p">))))</span>
<span class="w"> </span><span class="p">(</span><span class="nv">end</span><span class="w"> </span><span class="p">((</span><span class="k">lambda</span><span class="p">()</span>
<span class="w"> </span><span class="p">(</span><span class="nv">back-to-indentation</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nv">point</span><span class="p">)))))</span>
<span class="w"> </span><span class="p">(</span><span class="nv">kill-region</span><span class="w"> </span><span class="nv">beg</span><span class="w"> </span><span class="nv">end</span><span class="p">))</span>
<span class="w"> </span><span class="p">(</span><span class="nv">copy-symbol-at-point</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nv">insert</span><span class="w"> </span><span class="s">"iface->"</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nv">end-of-thing</span><span class="w"> </span><span class="ss">'symbol</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nv">delete-char</span><span class="w"> </span><span class="mi">1</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="k">let</span><span class="w"> </span><span class="p">((</span><span class="nv">beg</span><span class="w"> </span><span class="p">(</span><span class="nv">point</span><span class="p">))</span>
<span class="w"> </span><span class="p">(</span><span class="nv">end</span><span class="w"> </span><span class="p">((</span><span class="k">lambda</span><span class="p">()</span>
<span class="w"> </span><span class="p">(</span><span class="nv">find-list-end</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nv">point</span><span class="p">)))))</span>
<span class="w"> </span><span class="p">(</span><span class="nv">kill-region</span><span class="w"> </span><span class="nv">beg</span><span class="w"> </span><span class="nv">end</span><span class="p">))</span>
<span class="w"> </span><span class="p">(</span><span class="nv">insert</span><span class="w"> </span><span class="s">" = i_"</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nv">yank</span><span class="w"> </span><span class="mi">2</span><span class="p">))</span>
<span class="w"> </span><span class="p">(</span><span class="nv">next-line</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nv">beginning-of-line</span><span class="p">))</span>
<span class="p">(</span><span class="nb">defun</span><span class="w"> </span><span class="nv">implement-gobject-vfunc-prototype</span><span class="p">()</span>
<span class="w"> </span><span class="s">"Change a vtable line of a GObject interface to an implementation prototype line like:</span>
<span class="s">void (*my_iface_func)(type1 param1, type2 param2, ...);</span>
<span class="s">to</span>
<span class="s">static void</span>
<span class="s">i_my_iface_func(type1 param1, type2 param2, ...)"</span>
<span class="w"> </span><span class="p">(</span><span class="nv">interactive</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="k">let</span><span class="w"> </span><span class="p">((</span><span class="nv">beg</span><span class="w"> </span><span class="p">((</span><span class="k">lambda</span><span class="p">()</span>
<span class="w"> </span><span class="p">(</span><span class="nv">back-to-indentation</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nv">point</span><span class="p">))))</span>
<span class="w"> </span><span class="p">(</span><span class="nv">end</span><span class="w"> </span><span class="p">((</span><span class="k">lambda</span><span class="p">()</span>
<span class="w"> </span><span class="p">(</span><span class="nv">beginning-of-line</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nv">point</span><span class="p">)))))</span>
<span class="w"> </span><span class="p">(</span><span class="nv">kill-region</span><span class="w"> </span><span class="nv">beg</span><span class="w"> </span><span class="nv">end</span><span class="p">))</span>
<span class="w"> </span><span class="p">(</span><span class="nv">insert</span><span class="w"> </span><span class="s">"static "</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nv">search-forward</span><span class="w"> </span><span class="s">"(*"</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nv">delete-char</span><span class="w"> </span><span class="mi">-3</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nv">newline</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nv">insert</span><span class="w"> </span><span class="s">"i_"</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nv">end-of-thing</span><span class="w"> </span><span class="ss">'symbol</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nv">delete-char</span><span class="w"> </span><span class="mi">1</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="k">let</span><span class="w"> </span><span class="p">((</span><span class="nv">beg</span><span class="w"> </span><span class="p">(</span><span class="nv">point</span><span class="p">))</span>
<span class="w"> </span><span class="p">(</span><span class="nv">end</span><span class="w"> </span><span class="p">((</span><span class="k">lambda</span><span class="p">()</span>
<span class="w"> </span><span class="p">(</span><span class="nv">find-list-end</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nv">point</span><span class="p">)))))</span>
<span class="w"> </span><span class="p">(</span><span class="nv">indent-region</span><span class="w"> </span><span class="nv">beg</span><span class="w"> </span><span class="nv">end</span><span class="p">))</span>
<span class="w"> </span><span class="p">(</span><span class="nv">delete-char</span><span class="w"> </span><span class="mi">1</span><span class="p">))</span>
</pre></div>
<p>Now all I have to do is to copy the whole vtable entry into <tt class="docutils literal">matrix_http_api_matrix_api_init()</tt>,
execute <kbd>M-x implement-gobject-vfunc</kbd>, then put the same vtable entry somewhere before the
interface init function, and execute <kbd>M-x implement-gobject-vfunc-prototype</kbd>.</p>
How my e-mail gets to that other guy?2015-08-27T21:47:19+00:002015-08-27T21:47:19+00:00Gergely Polonkaitag:None,2015-08-27:2015/08/27/how-my-email-gets-to-that-other-guy/<p>A friend of mine asked me how it is possible that she pushes buttons on her keyboard and mouse,
and in an instant her peer reads the text she had in her mind. This is a step-by-step
introduction of what happens in-between.</p>
<div class="section" id="from-your-mind-to-your-computer">
<h2>From your mind to your computer</h2>
<p>When you …</p></div><p>A friend of mine asked me how it is possible that she pushes buttons on her keyboard and mouse,
and in an instant her peer reads the text she had in her mind. This is a step-by-step
introduction of what happens in-between.</p>
<div class="section" id="from-your-mind-to-your-computer">
<h2>From your mind to your computer</h2>
<p>When you decide to write an e-mail to an acquaintance of yours, you open up your mailing software
(this document doesn’t cover using mail applications you access through your browsers, just plain
old Thunderbird, Outlook or similar programs. However, it gets the same after the mail left your
computer), and press the “New Mail” button. What happens during this process is not covered in
this article, but feel free to ask me in a comment! Now that you have your Mail User Agent (<span class="caps">MUA</span>)
up and running, you begin typing.</p>
<p>When you press a button on your keyboard or mouse, a bunch of bits gets through the wire (or
through air, if you went wireless) and get into your computer. I guess you learned about Morse
during school; imagine two <a class="reference external" href="http://www.uscupstate.edu/academics/education/aam/lessons/susan_sawyer/morse%20code.jpg">Morse operators</a>, one
in your keyboard/mouse, and one in your computer. Whenever you press a key, that tiny creature
sends a series of short and long beeps (called 0 or 1 bits, respectively) to the operator in your
computer (fun fact: have you ever seen someone typing at an amazing speed of 5 key presses per
second? Now imagine that whenever that guy presses a key on their keyboard, that tiny little
Morse operator pressing his button 16 times for each key press, with perfect timing so that the
receiving operator can decide if that was a short or long beep.)</p>
<p>Now that the code got to the operator inside the machine, it’s up to him to decode it. The funny
thing about keyboards and computers is that the computer doesn’t receive the message “Letter Q was
pressed”, but instead “The second button on the second row was pressed” (a number called scan
code). At this time the operator decodes this information (in this example it is most likely this
Morse code: <tt class="docutils literal"><span class="pre">···-····</span> <span class="pre">-··-····</span></tt>) and checks one of his tables titled “Current Keyboard Layout.”
It says this specific key corresponds to letter ‘Q’, so it forwards this information (I mean the
letter; after this step your computer doesn’t care which plastic slab you hit, just the letter
‘Q’) to your <span class="caps">MUA</span>, inserts it into the mail in its memory, then displaying it happily (more about
this step later).</p>
<p>When you finish your letter you press the send button of your <span class="caps">MUA</span>. First it converts all the
pretty letters and pictures to something a computer can understand (yes, those Morse codes, or
more precisely, zeros and ones, again). Then it adds loads of meta data, like your name and
e-mail address, the current date and time including the time zone and pass it to the sending parts
of the <span class="caps">MUA</span> so the next step can begin.</p>
</div>
<div class="section" id="ip-addresses-dns-and-protocols">
<h2><span class="caps">IP</span> addresses, <span class="caps">DNS</span> and protocols</h2>
<p>The Internet is a huge amount of computers connected with each other, all of them having at least
one address called <span class="caps">IP</span> address that looks something like this: <tt class="docutils literal">123.234.112.221</tt>. These are four
numbers between 0 and 255 inclusive, separated by dots. This makes it possible to have
4,294,967,296 computers. With the rules of address assignment added, this is actually reduced to
3,702,258,432; a huge number, still, but it is not enough, as in the era of the Internet of Things
everything is interconnected, up to and possibly including your toaster. Thus, we are slowly
transitioning to a new addressing scheme that looks like this:
<tt class="docutils literal">1234:5678:90ab:dead:beef:9876:5432:1234</tt>. This gives an enormous amount of
340,282,366,920,938,463,463,374,607,431,768,211,456 addresses, with only
4,325,185,976,917,036,918,000,125,705,034,137,602 of them being reserved, which gives us only a
petty 335,957,180,944,021,426,545,374,481,726,734,073,854 available.</p>
<p>Imagine a large city with <a class="reference external" href="http://www.digitallifeplus.com/wp-content/uploads/2012/07/new-york-city-aerial-5.jpg">that many buildings</a>, all of
them having only a number: their <span class="caps">IP</span> address. No street names, no company names, no nothing. But
people tend to be bad at memorizing numbers, so they started to give these buildings names. For
example there is a house with the number <tt class="docutils literal">216.58.209.165</tt>, but between each other, people call
it <tt class="docutils literal">gmail.com</tt>. Much better, isn’t it? Unfortunately, when computers talk, they only
understand numbers so we have to provide them just that.</p>
<p>As remembering this huge number of addresses is a bit inconvenient, we created Domain Name
Service, or <span class="caps">DNS</span> for short. A “domain name” usually (but not always) consist of two strings of
letters, separated by dots (e.g. polonkai.eu, gmail.com, my-very-long-domain.co.uk, etc.), and a
hostname is a domain name occasionally prefixed with something (e.g. <strong>www</strong>.gmail.com,
<strong>my-server</strong>.my-very-long-domain.co.uk, etc.) One of the main jobs of <span class="caps">DNS</span> is to keep record of
hostname/address pairs. When you enter <tt class="docutils literal">gmail.com</tt> (which happens to be both a domain name and a
hostname) in your browser’s address bar, your computer asks the <span class="caps">DNS</span> service if it knows the actual
address of the building that people call <tt class="docutils literal">gmail.com</tt>. If it does, it will happily tell your
computer the number of that building.</p>
<p>Another <span class="caps">DNS</span> job is to store some meta data about these domain names. For such meta data there are
record types, one of these types being the Mail eXchanger, or <span class="caps">MX</span>. This record of a domain tells
the world who is handling incoming mails for the specified domain. For <tt class="docutils literal">gmail.com</tt> this is
<tt class="docutils literal"><span class="pre">gmail-smtp-in.l.google.com</span></tt> (among others; there can be multiple records of the same type, in
which case they usually have priorities, too.)</p>
<p>One more rule: when two computers talk to each other they use so called protocols. These
protocols define a set of rules on how they should communicate; this includes message formatting,
special code words and such.</p>
</div>
<div class="section" id="from-your-computer-to-the-mail-server">
<h2>From your computer to the mail server</h2>
<p>Your <span class="caps">MUA</span> has two settings called <span class="caps">SMTP</span> server address <span class="caps">SMTP</span> port number (see about that later).
<span class="caps">SMTP</span> stands for Simple Mail Transfer Protocol, and defines the rules on how your <span class="caps">MUA</span>, or another
mail handling computer should communicate with a mail handling computer when <em>sending</em> mail. Most
probably your Internet Service Provider gave you an <span class="caps">SMTP</span> server name, like <tt class="docutils literal">smtp.aol.com</tt> and a
port number like <tt class="docutils literal">587</tt>.</p>
<p>When you hit that send button of yours, your computer will check with the <span class="caps">DNS</span> service for the
address of the <span class="caps">SMTP</span> server, which, for <tt class="docutils literal">smtp.aol.com</tt>, is <tt class="docutils literal">64.12.88.133</tt>. The computer puts
this name/address pair into its memory, so it doesn’t have to ask the <span class="caps">DNS</span> again (this technique is
called caching and is widely used wherever time consuming operations happen).</p>
<p>Then it will send your message to the given port number of this newly fetched address. If you
imagined computers as office buildings, you can imagine port numbers as departments and there can
be 65535 of them in one building. The port number of <span class="caps">SMTP</span> is usually 25, 465 or 587 depending on
many things we don’t cover here. Your <span class="caps">MUA</span> prepares your letter, adding your e-mail address and
the recipients’, together with other information that may be useful for transferring your mail.
It then puts this well formatted message in an envelope and writes “to building <tt class="docutils literal">64.12.88.133</tt>,
dept. <tt class="docutils literal">587</tt>”, and puts it on the wire so it gets there (if the wire is broken, the building does
not exist or there is no such department, you will get an error message from your <span class="caps">MUA</span>). Your
address and the recipient’s address are inside the envelope; other than the <span class="caps">MUA</span>, your own computer
is not concerned about it.</p>
<p>The mailing department (or instead lets call it the Mail Transfer Agent, <span class="caps">A.K.A.</span> <span class="caps">MTA</span>) now opens
this envelope and reads the letter. All of it, letter by letter, checking if your <span class="caps">MUA</span> formatted
it well. More than probably it also runs your message through several filters to decide if you
are a bad guy sending some unwanted letter (also known as spam), but most importantly it fetches
the recipients address. It is possible, e.g. when you send an e-mail within the same
organization, that the recipient’s address is handled by this very same computer. In this case
the <span class="caps">MTA</span> puts the mail to the recipient’s mailbox and the next step is skipped.</p>
</div>
<div class="section" id="from-one-server-to-another">
<h2>From one server to another</h2>
<p>Naturally, it is possible to send an e-mail from one company to another, so these MTAs don’t just
wait for e-mails from you, but also communicate with each other. When you send a letter from your
<tt class="docutils literal">example@aol.com</tt> address to me at <tt class="docutils literal">gergely@polonkai.eu</tt>, this is what happens.</p>
<p>In this case, the <span class="caps">MTA</span> that initially received the e-mail from you (which happened to be your
Internet Service Provider’s <span class="caps">SMTP</span> server) turns to the <span class="caps">DNS</span> again. It will ask for the <span class="caps">MX</span> record of
the domain name specified by the e-mail address, (the part after the <tt class="docutils literal">@</tt> character, in my case,
<tt class="docutils literal">polonkai.eu</tt>), because the server mentioned that must be contacted, so they can deliver your
mail for me. My domain is configured so its primary <span class="caps">MX</span> record is <tt class="docutils literal">aspmx.l.google.com</tt> and the
secondary is <tt class="docutils literal">alt1.aspmx.l.google.com</tt> (and 5 more. Google likes to play it safe.) The <span class="caps">MTA</span>
then gets the first server name, asks the <span class="caps">DNS</span> for its address, and tries to send a message to the
<tt class="docutils literal">173.194.67.27</tt> (the address of <tt class="docutils literal">aspmx.l.google.com</tt>), same department. But unlike your <span class="caps">MUA</span>,
MTAs don’t have a pre-defined port number for other MTAs (although there can be exceptions).
Instead, they use well-known port numbers, <tt class="docutils literal">465</tt> and <tt class="docutils literal">25</tt>. If the <span class="caps">MTA</span> on that server cannot
be contacted for any reason, it tries the next one on the list of <span class="caps">MX</span> records. If none of the
servers can be contacted, it will retry based on a set of rules defined by the administrators,
which usually means it will retry after 1, 4, 24 and 48 hours. If there is still no answer after
that many attempts, you will get an error message back, in the form of an e-mail sent directly by
the <span class="caps">SMTP</span> server.</p>
<p>Once the other <span class="caps">MTA</span> could be contacted, your message is sent there. The original envelope you used
is discarded, and a new one is used with the address and dept. number (port) of the receiving <span class="caps">MTA</span>.
Also, your message gets altered a little bit, as most MTAs are kind enough (ie. not sneaky) to add
a clause to your message stating “the <span class="caps">MTA</span> at <organization> has checked and forwarded this message.”</p>
<p>It is possible, though not likely, that your message gets through more than two MTAs (one at your
<span class="caps">ISP</span> and one at the receiver’s) before arriving to its destination. At the end, an <span class="caps">MTA</span> will say
that “<span class="caps">OK</span>, this recipient address is handled by me”, your message stops and stays there, put in
your peer’s mailbox.</p>
<div class="section" id="the-mailbox">
<h3>The mailbox</h3>
<p>Now that the <span class="caps">MTA</span> has passed your mail to the mailbox team (I call it a team instead of department
because the tasks described here are usually handled by the <span class="caps">MTA</span>, too), it reads it. (Pesky little
guys these mail handling departments, aren’t they?) If the mailbox has some filtering rules, like
“if <span class="caps">XY</span> sends me a letter, mark it as important” or “if the letter has a specific word in its
subject, put it in the <span class="caps">XY</span> folder”, it executes them, but the main point is to land the message in
the actual post box of the recipient.</p>
</div>
</div>
<div class="section" id="from-the-post-box-to-the-recipients-computer">
<h2>From the post box to the recipients computer</h2>
<p>When the recipient opens their <span class="caps">MUA</span>, it will look to a setting usually called “Incoming mail
server”. Just like the <span class="caps">SMTP</span> server, it has a name and port number, along with a server type.
This type can vary from provider to provider, and is usually one of <span class="caps">POP3</span> (pretty old protocol,
doesn’t even support folders on its own), <span class="caps">IMAP</span> (a newer one, with folders and message flags like
“important”), <span class="caps">MAPI</span> (a dialect of <span class="caps">IMAP</span>, created by Microsoft as far as I know), or plain old mbox
files on the receiving computer (this last option is pretty rare nowadays, so I don’t cover this
option. Also, if you use these, you most probably don’t really need this article to understand
how these things work.) This latter setting defines the protocol, telling your <span class="caps">MUA</span> how to “speak”
to the post box.</p>
<p>So your <span class="caps">MUA</span> turns to the <span class="caps">DNS</span> once more to get the address of your incoming mail server and
contacts it, using the protocol set by the server type. At the end, the recipients computer will
receive a bunch of envelopes including the one that contains your message. The <span class="caps">MUA</span> opens them one
by one and reads them, making a list ordered by their sender or subject, or the date of sending.</p>
</div>
<div class="section" id="from-the-recipients-comupter-to-their-eyes">
<h2>From the recipient’s comupter to their eyes</h2>
<p>When the recipient then clicks on one of these mails, the <span class="caps">MUA</span> will fetch all the relevant bits
like the sender, the subject line, the date of sending and the contents itself and sends it to the
“printing” department (I use quotes as they don’t really print your mail on paper, they just
convert it to a nice image so the recipient can see it. This is sometimes referred to as a
rendering engine). Based on a bunch of rules they pretty-print it and send it to your display as
a new series of Morse codes. Your display then decides how it will present it to the user: draw
the pretty pictures if it is a computer screen, or just raise and lower some hard dots that
represents letters on a Braille terminal.</p>
</div>
F/OSS Fail meter2015-08-19T10:12:19+00:002015-08-19T10:12:19+00:00Gergely Polonkaitag:None,2015-08-19:2015/08/19/foss-failmeter/<p>I have recently bumped into <a class="reference external" href="http://spot.livejournal.com/308370.html">this article</a>. Naturally,
I quickly calculated the <span class="caps">FAIL</span> metrics for all my projects (most of them are pretty high). To ease
calculation, I made up a <a class="reference external" href="{static}../failmeter/index.html">small page</a> based on this list
(although I have divided the points by 5; I really don’t understand …</p><p>I have recently bumped into <a class="reference external" href="http://spot.livejournal.com/308370.html">this article</a>. Naturally,
I quickly calculated the <span class="caps">FAIL</span> metrics for all my projects (most of them are pretty high). To ease
calculation, I made up a <a class="reference external" href="{static}../failmeter/index.html">small page</a> based on this list
(although I have divided the points by 5; I really don’t understand why spot is using such big
points if all of them can be divided by 5). Feel free to use it, and if you have any
recommendations (point additions/removal, new categories, etc.), leave me a comment!</p>
@ParamConverter à la Django2015-06-07T18:14:32+00:002015-06-07T18:14:32+00:00Gergely Polonkaitag:None,2015-06-07:2015/06/07/paramconverter-a-la-django/<p>One thing I really miss from <a class="reference external" href="https://www.djangoproject.com/">Django</a> is <a class="reference external" href="http://symfony.com/">Symfony</a>’s <a class="reference external" href="http://symfony.com/doc/current/bundles/SensioFrameworkExtraBundle/annotations/converters.html">@ParamConverter</a>. It
made my life so much easier while developing with Symfony. In Django, of course, there is
<a class="reference external" href="https://docs.djangoproject.com/en/dev/topics/http/shortcuts/#get-object-or-404">get_object_or_404</a> but, for
example, in one of my projects I had a view that had to resolve 6(!) objects from the <span class="caps">URL …</span></p><p>One thing I really miss from <a class="reference external" href="https://www.djangoproject.com/">Django</a> is <a class="reference external" href="http://symfony.com/">Symfony</a>’s <a class="reference external" href="http://symfony.com/doc/current/bundles/SensioFrameworkExtraBundle/annotations/converters.html">@ParamConverter</a>. It
made my life so much easier while developing with Symfony. In Django, of course, there is
<a class="reference external" href="https://docs.djangoproject.com/en/dev/topics/http/shortcuts/#get-object-or-404">get_object_or_404</a> but, for
example, in one of my projects I had a view that had to resolve 6(!) objects from the <span class="caps">URL</span>, and
writing <tt class="docutils literal">get_object_or_404</tt> six times is not what a programmer likes to do (yes, this view had a
refactor later on). A quick Google search gave me one <a class="reference external" href="http://openclassrooms.com/forum/sujet/middleware-django-genre-paramconverter-doctrine">usable result</a> (in
French), but it was very generalized that I cannot always use. Also, it was using a middleware,
which may introduce performance issues sometimes <sup>[citation needed]</sup>. So I decided to go
with decorators, and at the end, I came up with this:</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">re</span>
<span class="kn">from</span> <span class="nn">django.shortcuts</span> <span class="kn">import</span> <span class="n">get_object_or_404</span>
<span class="kn">from</span> <span class="nn">django.db</span> <span class="kn">import</span> <span class="n">models</span>
<span class="k">def</span> <span class="nf">convert_params</span><span class="p">(</span><span class="o">*</span><span class="n">params_to_convert</span><span class="p">,</span> <span class="o">**</span><span class="n">options</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""</span>
<span class="sd"> Convert parameters to objects. Each parameter to this decorator</span>
<span class="sd"> must be a model instance (subclass of django.db.models.Model) or a</span>
<span class="sd"> tuple with the following members:</span>
<span class="sd"> * model: a Model subclass</span>
<span class="sd"> * param_name: the name of the parameter that holds the value to be</span>
<span class="sd"> matched. If not exists, or is None, the model’s class name will</span>
<span class="sd"> be converted from ModelName to model_name form, suffixed with</span>
<span class="sd"> "_id". E.g. for MyModel, the default will be my_model_id</span>
<span class="sd"> * the field name against which the value in param_name will be</span>
<span class="sd"> matched. If not exists or is None, the default will be "id"</span>
<span class="sd"> * obj_param_name: the name of the parameter that will hold the</span>
<span class="sd"> resolved object. If not exists or None, the default value will</span>
<span class="sd"> be the model’s class name converted from ModelName to model_name</span>
<span class="sd"> form, e.g. for MyModel, the default value will be my_model.</span>
<span class="sd"> The values are resolved with get_object_or_404, so if the given</span>
<span class="sd"> object doesn’t exist, it will redirect to a 404 page. If you want</span>
<span class="sd"> to allow non-existing models, pass prevent_404=True as a keyword</span>
<span class="sd"> argument.</span>
<span class="sd"> """</span>
<span class="n">prevent_404</span> <span class="o">=</span> <span class="n">options</span><span class="o">.</span><span class="n">pop</span><span class="p">(</span><span class="s1">'prevent_404'</span><span class="p">,</span> <span class="kc">False</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">is_model</span><span class="p">(</span><span class="n">m</span><span class="p">):</span>
<span class="k">return</span> <span class="nb">issubclass</span><span class="p">(</span><span class="nb">type</span><span class="p">(</span><span class="n">m</span><span class="p">),</span> <span class="n">models</span><span class="o">.</span><span class="n">base</span><span class="o">.</span><span class="n">ModelBase</span><span class="p">)</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">params_to_convert</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s2">"Must pass at least one parameter spec!"</span><span class="p">)</span>
<span class="k">if</span> <span class="p">(</span>
<span class="nb">len</span><span class="p">(</span><span class="n">params_to_convert</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span> <span class="ow">and</span> \
<span class="nb">hasattr</span><span class="p">(</span><span class="n">params_to_convert</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="s1">'__call__'</span><span class="p">)</span> <span class="ow">and</span> \
<span class="ow">not</span> <span class="n">is_model</span><span class="p">(</span><span class="n">params_to_convert</span><span class="p">[</span><span class="mi">0</span><span class="p">])):</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s2">"This decorator must have arguments!"</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">convert_params_decorator</span><span class="p">(</span><span class="n">func</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">wrapper</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="n">converted_params</span> <span class="o">=</span> <span class="p">()</span>
<span class="k">for</span> <span class="n">pspec</span> <span class="ow">in</span> <span class="n">params_to_convert</span><span class="p">:</span>
<span class="c1"># If the current pspec is not a tuple, let’s assume</span>
<span class="c1"># it’s a model class</span>
<span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">pspec</span><span class="p">,</span> <span class="nb">tuple</span><span class="p">):</span>
<span class="n">pspec</span> <span class="o">=</span> <span class="p">(</span><span class="n">pspec</span><span class="p">,)</span>
<span class="c1"># First, and the only required element in the</span>
<span class="c1"># parameters is the model name which this object</span>
<span class="c1"># belongs to</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">pspec</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">is_model</span><span class="p">(</span><span class="n">model</span><span class="p">):</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span>
<span class="s2">"First value in pspec must be a Model subclass!"</span><span class="p">)</span>
<span class="c1"># We will calculate these soon…</span>
<span class="n">param_name</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">calc_obj_name</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">sub</span><span class="p">(</span>
<span class="s1">'([a-z0-9])([A-Z])'</span><span class="p">,</span>
<span class="sa">r</span><span class="s1">'\1_\2'</span><span class="p">,</span>
<span class="n">re</span><span class="o">.</span><span class="n">sub</span><span class="p">(</span>
<span class="s1">'(.)([A-Z][a-z]+)'</span><span class="p">,</span>
<span class="sa">r</span><span class="s1">'\1_\2'</span><span class="p">,</span>
<span class="n">model</span><span class="o">.</span><span class="vm">__name__</span><span class="p">))</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span>
<span class="n">obj_field_name</span> <span class="o">=</span> <span class="kc">None</span>
<span class="c1"># The second element, if not None, is the keyword</span>
<span class="c1"># parameter name that holds the value to convert</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">pspec</span><span class="p">)</span> <span class="o"><</span> <span class="mi">2</span> <span class="ow">or</span> <span class="n">pspec</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="n">param_name</span> <span class="o">=</span> <span class="n">calc_obj_name</span> <span class="o">+</span> <span class="s1">'_id'</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">param_name</span> <span class="o">=</span> <span class="n">pspec</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="k">if</span> <span class="n">param_name</span> <span class="ow">in</span> <span class="n">converted_params</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s1">'</span><span class="si">%s</span><span class="s1"> is already converted'</span> <span class="o">%</span> <span class="n">param_name</span><span class="p">)</span>
<span class="n">converted_params</span> <span class="o">+=</span> <span class="p">(</span><span class="n">param_name</span><span class="p">,)</span>
<span class="n">field_value</span> <span class="o">=</span> <span class="n">kwargs</span><span class="o">.</span><span class="n">pop</span><span class="p">(</span><span class="n">param_name</span><span class="p">)</span>
<span class="c1"># The third element is the field name which must be</span>
<span class="c1"># equal to the specified value. If it doesn’t exist or</span>
<span class="c1"># None, it defaults to 'id'</span>
<span class="k">if</span> <span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">pspec</span><span class="p">)</span> <span class="o"><</span> <span class="mi">3</span><span class="p">)</span> <span class="ow">or</span> <span class="n">pspec</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="n">obj_field_name</span> <span class="o">=</span> <span class="s1">'id'</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">obj_field_name</span> <span class="o">=</span> <span class="n">pspec</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span>
<span class="c1"># The fourth element is the parameter name for the</span>
<span class="c1"># object. If the parameter already exists, we consider</span>
<span class="c1"># it an error</span>
<span class="k">if</span> <span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">pspec</span><span class="p">)</span> <span class="o"><</span> <span class="mi">4</span><span class="p">)</span> <span class="ow">or</span> <span class="n">pspec</span><span class="p">[</span><span class="mi">3</span><span class="p">]</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="n">obj_param_name</span> <span class="o">=</span> <span class="n">calc_obj_name</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">obj_param_name</span> <span class="o">=</span> <span class="n">pspec</span><span class="p">[</span><span class="mi">3</span><span class="p">]</span>
<span class="k">if</span> <span class="n">obj_param_name</span> <span class="ow">in</span> <span class="n">kwargs</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">KeyError</span><span class="p">(</span>
<span class="s2">"'</span><span class="si">%s</span><span class="s2">' already exists as a parameter"</span> <span class="o">%</span> <span class="n">obj_param_name</span><span class="p">)</span>
<span class="n">filter_kwargs</span> <span class="o">=</span> <span class="p">{</span><span class="n">obj_field_name</span><span class="p">:</span> <span class="n">field_value</span><span class="p">}</span>
<span class="k">if</span> <span class="p">(</span><span class="n">prevent_404</span><span class="p">):</span>
<span class="n">kwargs</span><span class="p">[</span><span class="n">obj_param_name</span><span class="p">]</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span>
<span class="o">**</span><span class="n">filter_kwargs</span><span class="p">)</span><span class="o">.</span><span class="n">first</span><span class="p">()</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">kwargs</span><span class="p">[</span><span class="n">obj_param_name</span><span class="p">]</span> <span class="o">=</span> <span class="n">get_object_or_404</span><span class="p">(</span>
<span class="n">model</span><span class="p">,</span>
<span class="o">**</span><span class="n">filter_kwargs</span><span class="p">)</span>
<span class="k">return</span> <span class="n">func</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="k">return</span> <span class="n">wrapper</span>
<span class="k">return</span> <span class="n">convert_params_decorator</span>
</pre></div>
<p>Now I can decorate my views, either class or function based, with <tt class="docutils literal">@convert_params(User,
(Article, <span class="pre">'aid'),</span> (Paragraph, None, <span class="pre">'pid'),</span> (AnotherObject, None, None, <span class="pre">'obj'))</span></tt> and all the
magic happens in the background. The <tt class="docutils literal">user_id</tt> parameter passed to my function will be popped
off, and be resolved against the <tt class="docutils literal">User</tt> model by using the <tt class="docutils literal">id</tt> field; the result is put in
the new <tt class="docutils literal">user</tt> parameter. For Article, the <tt class="docutils literal">aid</tt> parameter will be matched against the <tt class="docutils literal">id</tt>
field of the <tt class="docutils literal">Article</tt> model putting the result into <tt class="docutils literal">article</tt>, and finally, the
<tt class="docutils literal">another_object_id</tt> will be matched against the <tt class="docutils literal">id</tt> field of the <tt class="docutils literal">AnotherObject</tt> model, but
in this case, the result is passed to the original function as <tt class="docutils literal">obj</tt>.</p>
Cross browser border-radius SASS mixin with varargs2015-04-27T22:59:56+00:002015-04-27T22:59:56+00:00Gergely Polonkaitag:None,2015-04-27:blog/2015/4/28/cross-browser-border-radius-sass-mixin-with-varargs.html<p>Few days ago I needed to create style sheets with many rounded boxes, where different corners had
to be rounded differently (think about Bootstrap’s <a class="reference external" href="http://getbootstrap.com/components/#btn-groups">button groups</a>).</p>
<p><span class="caps">CSS</span> has this nifty shorthand to specify border width in one line, like with <tt class="docutils literal"><span class="pre">border-width:</span> 1px
2px 3px 4px</tt>, but it lacks the …</p><p>Few days ago I needed to create style sheets with many rounded boxes, where different corners had
to be rounded differently (think about Bootstrap’s <a class="reference external" href="http://getbootstrap.com/components/#btn-groups">button groups</a>).</p>
<p><span class="caps">CSS</span> has this nifty shorthand to specify border width in one line, like with <tt class="docutils literal"><span class="pre">border-width:</span> 1px
2px 3px 4px</tt>, but it lacks the same for <tt class="docutils literal"><span class="pre">border-radius</span></tt>. So I decided to create something
similar using <a class="reference external" href="http://sass-lang.com/guide#topic-6">Sass mixins</a> with dynamic parameters.
Another nice feature you get using the <tt class="docutils literal"><span class="pre">border-width</span></tt> shorthand is that you can specify less
than four parameters, and the values will be applied on different sides of your box, so in the end
all side will have the whole <tt class="docutils literal"><span class="pre">border-width</span></tt> set.</p>
<p>I wanted to achieve the same for my <tt class="docutils literal"><span class="pre">border-radius</span></tt> mixin, although I
could not start specifically with the <cite>top</cite> side. I decided to go with
the top right corner for the first parameter, while trying to keep a
sane repeating pattern. Here is the result:</p>
<div class="highlight"><pre><span></span><span class="nf">=border-width</span><span class="p">(</span><span class="nv">$t</span><span class="o">,</span><span class="w"> </span><span class="nv">$r</span><span class="o">:</span><span class="w"> </span><span class="nv">$t</span><span class="o">,</span><span class="w"> </span><span class="nv">$b</span><span class="o">:</span><span class="w"> </span><span class="nv">$t</span><span class="o">,</span><span class="w"> </span><span class="nv">$l</span><span class="o">:</span><span class="w"> </span><span class="nv">$r</span><span class="p">)</span>
<span class="w"> </span><span class="na">border-top-width</span><span class="o">:</span><span class="w"> </span><span class="nv">$t</span>
<span class="w"> </span><span class="na">border-right-width</span><span class="o">:</span><span class="w"> </span><span class="nv">$r</span>
<span class="w"> </span><span class="na">border-bottom-width</span><span class="o">:</span><span class="w"> </span><span class="nv">$b</span>
<span class="w"> </span><span class="na">border-left-width</span><span class="o">:</span><span class="w"> </span><span class="nv">$l</span>
<span class="nf">=border-top-right-radius</span><span class="p">(</span><span class="nv">$value</span><span class="p">)</span>
<span class="w"> </span><span class="na">border-top-right-radius</span><span class="o">:</span><span class="w"> </span><span class="nv">$value</span>
<span class="w"> </span><span class="na">-moz-border-top-right-radius</span><span class="o">:</span><span class="w"> </span><span class="nv">$value</span>
<span class="w"> </span><span class="na">-webkit-border-top-right-radius</span><span class="o">:</span><span class="w"> </span><span class="nv">$value</span>
<span class="nf">=border-top-left-radius</span><span class="p">(</span><span class="nv">$value</span><span class="p">)</span>
<span class="w"> </span><span class="na">border-top-left-radius</span><span class="o">:</span><span class="w"> </span><span class="nv">$value</span>
<span class="w"> </span><span class="na">-moz-border-top-left-radius</span><span class="o">:</span><span class="w"> </span><span class="nv">$value</span>
<span class="w"> </span><span class="na">-webkit-border-top-left-radius</span><span class="o">:</span><span class="w"> </span><span class="nv">$value</span>
<span class="nf">=border-bottom-right-radius</span><span class="p">(</span><span class="nv">$value</span><span class="p">)</span>
<span class="w"> </span><span class="na">border-bottom-right-radius</span><span class="o">:</span><span class="w"> </span><span class="nv">$value</span>
<span class="w"> </span><span class="na">-moz-border-bottom-right-radius</span><span class="o">:</span><span class="w"> </span><span class="nv">$value</span>
<span class="w"> </span><span class="na">-webkit-border-bottom-right-radius</span><span class="o">:</span><span class="w"> </span><span class="nv">$value</span>
<span class="nf">=border-bottom-left-radius</span><span class="p">(</span><span class="nv">$value</span><span class="p">)</span>
<span class="w"> </span><span class="na">border-bottom-left-radius</span><span class="o">:</span><span class="w"> </span><span class="nv">$value</span>
<span class="w"> </span><span class="na">-moz-border-bottom-left-radius</span><span class="o">:</span><span class="w"> </span><span class="nv">$value</span>
<span class="w"> </span><span class="na">-webkit-border-bottom-left-radius</span><span class="o">:</span><span class="w"> </span><span class="nv">$value</span>
<span class="nf">=border-radius</span><span class="p">(</span><span class="nv">$tr</span><span class="o">,</span><span class="w"> </span><span class="nv">$br</span><span class="o">:</span><span class="w"> </span><span class="nv">$tr</span><span class="o">,</span><span class="w"> </span><span class="nv">$bl</span><span class="o">:</span><span class="w"> </span><span class="nv">$br</span><span class="o">,</span><span class="w"> </span><span class="nv">$tl</span><span class="o">:</span><span class="w"> </span><span class="nv">$tr</span><span class="p">)</span>
<span class="w"> </span><span class="nd">+border-top-right-radius</span><span class="p">(</span><span class="nv">$tr</span><span class="p">)</span>
<span class="w"> </span><span class="nd">+border-bottom-right-radius</span><span class="p">(</span><span class="nv">$br</span><span class="p">)</span>
<span class="w"> </span><span class="nd">+border-bottom-left-radius</span><span class="p">(</span><span class="nv">$bl</span><span class="p">)</span>
<span class="w"> </span><span class="nd">+border-top-left-radius</span><span class="p">(</span><span class="nv">$tl</span><span class="p">)</span>
</pre></div>
Good bye, Digital Ocean! Hello again, GitHub!2015-04-25T21:18:56+00:002015-04-25T21:18:56+00:00Gergely Polonkaitag:None,2015-04-25:blog/2015/4/25/good-bye-digital-ocean-hello-again-github.html<p>Few years ago I have signed up for a <a class="reference external" href="https://www.digitalocean.com/">Digital Ocean</a> account. I
used one single droplet for my private needs, like hosting my private Git repositories and my
blog. However, as I didn’t host anything else there except my blog, I decided to shut it down.
From now …</p><p>Few years ago I have signed up for a <a class="reference external" href="https://www.digitalocean.com/">Digital Ocean</a> account. I
used one single droplet for my private needs, like hosting my private Git repositories and my
blog. However, as I didn’t host anything else there except my blog, I decided to shut it down.
From now on, my blog is on <a class="reference external" href="https://pages.github.com/">GitHub Pages</a>, as it provides just
everything I need (except automatically converting my resume to <span class="caps">PDF</span>. But I can live without that.)</p>
<p>I’m really sorry, Digital Ocean Guys, your hosting is awesome and I’ll keep recommending you to
others, but paying for a droplet for one single blog is overkill.</p>
Using Git bisect to find the first good commit2015-02-26T10:42:56+00:002015-02-26T10:42:56+00:00Gergely Polonkaitag:None,2015-02-26:blog/2015/2/26/using-git-bisect-to-find-the-first-good-commit.html<p>Few months ago we “implemented” a bug in our software, which was released to the customers. We
continued development for two weeks when the first customer ticket arrived about the bug. We
successfully reproduced it with the customer’s version, but not with the development sources; it
turned out that …</p><p>Few months ago we “implemented” a bug in our software, which was released to the customers. We
continued development for two weeks when the first customer ticket arrived about the bug. We
successfully reproduced it with the customer’s version, but not with the development sources; it
turned out that one of the developers unconsciously fixed the bug. The devs spent some hours
finding where the fix lied before coming to me like “There is <tt class="docutils literal"><span class="pre">git-bisect</span></tt> which we can use to
find the commit where we messed up things. Is there a way to find where we fixed it?”</p>
<p>For those who don’t know this feature, you have to mark a known “good” and “bad” commit, then
git-bisect will go through the commits between this two, present you the corresponding snapshots,
and you have to mark each of them as “good” or “bad”. At the end, you will get a commit hash
where the bug first occured.</p>
<p>As it turned out, our developers’ problem rooted in the naming convention of git-bisect: they
assumed that the “good” commit must be a working one, while a “bad” one must be the buggy. In
this case, we did the following:</p>
<p>The commit with the customer’s release tag was marked as good (even though this had the bug), and
the latest commit on our development branch was marked as “bad” (even though the bug was fixed by
then). Now with every snapshot presented by git-bisect we had to do the opposite what you usually
do: mark commits still having the bug as “good”, and commits that don’t as “bad”. At the end, we
had the hash of the commit that fixed the bug (among some other things; luckily, the developer who
pushed that commit had a workflow that introduced a lot of cherry-picking and squashing before the
push, so he could easily find the bit that actually fixed the problem in his local repository with
the same technique).</p>
<p><a class="reference external" href="http://stackoverflow.com/a/17153598/1305139">This StackOverflow answer</a> suggests the very same,
but with some aliases:</p>
<div class="highlight"><pre><span></span><span class="k">[alias]</span>
<span class="w"> </span><span class="na">bisect-fixed</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">bisect bad</span>
<span class="w"> </span><span class="na">bisect-unfixed</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">bisect good</span>
</pre></div>
Rounding numbers to N decimals in Emacs2014-10-07T10:28:50+00:002014-10-07T10:28:50+00:00Gergely Polonkaitag:None,2014-10-07:blog/2014/10/7/rounding-numbers-to-n-decimals-in-emacs.html<p>I have recently faced a problem, where I had a bunch of <span class="caps">SVG</span> files with a large amount of fraction
numbers in the path definitions. These images were displayed in small size, so this amount of
precision was irrelevant, and these numbers took almost half of my <span class="caps">SVG</span> images’ size …</p><p>I have recently faced a problem, where I had a bunch of <span class="caps">SVG</span> files with a large amount of fraction
numbers in the path definitions. These images were displayed in small size, so this amount of
precision was irrelevant, and these numbers took almost half of my <span class="caps">SVG</span> images’ size. So I created
an Elisp defun to round these numbers to 2 decimals:</p>
<div class="highlight"><pre><span></span><span class="p">(</span><span class="nb">defun</span><span class="w"> </span><span class="nv">get-number-at-point</span><span class="w"> </span><span class="p">()</span>
<span class="w"> </span><span class="p">(</span><span class="nv">interactive</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nv">skip-chars-backward</span><span class="w"> </span><span class="s">"0123456789.-"</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nb">or</span><span class="w"> </span><span class="p">(</span><span class="nv">looking-at</span><span class="w"> </span><span class="s">"[0123456789.-]+"</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nb">error</span><span class="w"> </span><span class="s">"No number at point"</span><span class="p">))</span>
<span class="w"> </span><span class="p">(</span><span class="nv">string-to-number</span><span class="w"> </span><span class="p">(</span><span class="nv">match-string</span><span class="w"> </span><span class="mi">0</span><span class="p">)))</span>
<span class="p">(</span><span class="nb">defun</span><span class="w"> </span><span class="nv">round-number-at-point-to-decimals</span><span class="w"> </span><span class="p">(</span><span class="nv">decimal-count</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="nv">interactive</span><span class="w"> </span><span class="s">"NDecimal count: "</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="k">let</span><span class="w"> </span><span class="p">((</span><span class="nv">mult</span><span class="w"> </span><span class="p">(</span><span class="nb">expt</span><span class="w"> </span><span class="mi">10</span><span class="w"> </span><span class="nv">decimal-count</span><span class="p">)))</span>
<span class="w"> </span><span class="p">(</span><span class="nv">replace-match</span><span class="w"> </span><span class="p">(</span><span class="nv">number-to-string</span>
<span class="w"> </span><span class="p">(</span><span class="nb">/</span>
<span class="w"> </span><span class="p">(</span><span class="nb">fround</span>
<span class="w"> </span><span class="p">(</span><span class="nb">*</span>
<span class="w"> </span><span class="nv">mult</span>
<span class="w"> </span><span class="p">(</span><span class="nv">get-number-at-point</span><span class="p">)))</span>
<span class="w"> </span><span class="nv">mult</span><span class="p">)))))</span>
</pre></div>
<p>This finds the first digit of the number under point (the cursor), and reduces its digits to the
given amount (or the number given with <kbd>C-u</kbd>). It has some drawbacks, though, as it cannot
handle exponential forms (e.g. <tt class="docutils literal"><span class="pre">1e-1234</span></tt>), but these were rare in my case, and its hard to
iterate through all numbers. I will come over this latter problem soon(ish).</p>
NyanMacs2014-09-17T12:45:42+00:002014-09-17T12:45:42+00:00Gergely Polonkaitag:None,2014-09-17:blog/2014/9/17/nyanmacs.html<p>I was a Vi/ViM user for years. For several reasons I had to change to Emacs now and then. And
then, I found <a class="reference external" href="https://github.com/TeMPOraL/nyan-mode/">this</a>. I surrender. Emacs is just
better. (And this addon is working even in plain text mode without graphics)</p>
Registering an enum type in GLib, glib-mkenums magic2014-08-16T15:10:54+00:002014-08-16T15:10:54+00:00Gergely Polonkaitag:None,2014-08-16:blog/2014/8/16/registering-an-enum-type-in-glib-glib-mkenums-magic.html<p>In <a class="reference external" href="blog/2013/1/6/registering-an-enum-type-in-glib-s-type-system.html">this post</a> I said I
will get through the GLib Makefiles to add an enum type to GLib in a more sophisticated way.</p>
<p>In my other project, <a class="reference external" href="https://github.com/gergelypolonkai/swe-glib"><span class="caps">SWE</span>-GLib</a> I already used this
method. The following two rules in <tt class="docutils literal">Makefile.am</tt> create <tt class="docutils literal"><span class="pre">gswe-enumtypes.h</span></tt> and
<tt class="docutils literal"><span class="pre">gswe-enumtypes.c</span></tt>.</p>
<div class="highlight"><pre><span></span><span class="nv">gswe_enum_headers</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>headers-that-contain-enums …</pre></div><p>In <a class="reference external" href="blog/2013/1/6/registering-an-enum-type-in-glib-s-type-system.html">this post</a> I said I
will get through the GLib Makefiles to add an enum type to GLib in a more sophisticated way.</p>
<p>In my other project, <a class="reference external" href="https://github.com/gergelypolonkai/swe-glib"><span class="caps">SWE</span>-GLib</a> I already used this
method. The following two rules in <tt class="docutils literal">Makefile.am</tt> create <tt class="docutils literal"><span class="pre">gswe-enumtypes.h</span></tt> and
<tt class="docutils literal"><span class="pre">gswe-enumtypes.c</span></tt>.</p>
<div class="highlight"><pre><span></span><span class="nv">gswe_enum_headers</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>headers-that-contain-enums.h
<span class="nf">gswe-enumtypes.h</span><span class="o">:</span><span class="w"> </span><span class="k">$(</span><span class="nv">gswe_enum_headers</span><span class="k">)</span> <span class="n">gswe</span>-<span class="n">enumtypes</span>.<span class="n">h</span>.<span class="n">template</span>
<span class="w"> </span><span class="k">$(</span>GLIB_MKENUMS<span class="k">)</span><span class="w"> </span>--template<span class="w"> </span><span class="k">$(</span>filter<span class="w"> </span>%.template,$^<span class="k">)</span><span class="w"> </span><span class="k">$(</span>filter-out<span class="w"> </span>%.template,$^<span class="k">)</span><span class="w"> </span>><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>gswe-enumtypes.h.tmp<span class="w"> </span><span class="o">&&</span><span class="w"> </span>mv<span class="w"> </span>gswe-enumtypes.h.tmp<span class="w"> </span>gswe-enumtypes.h
<span class="nf">gswe-enumtypes.c</span><span class="o">:</span><span class="w"> </span><span class="k">$(</span><span class="nv">gswe_enum_headers</span><span class="k">)</span> <span class="n">gswe</span>-<span class="n">enumtypes</span>.<span class="n">h</span> <span class="n">gswe</span>-<span class="n">enumtypes</span>.<span class="n">c</span>.<span class="n">template</span>
<span class="w"> </span><span class="k">$(</span>GLIB_MKENUMS<span class="k">)</span><span class="w"> </span>--template<span class="w"> </span><span class="k">$(</span>filter<span class="w"> </span>%.template,$^<span class="k">)</span><span class="w"> </span><span class="k">$(</span>filter-out<span class="w"> </span>%.template,$^<span class="k">)</span><span class="w"> </span>><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>gswe-enumtypes.c.tmp<span class="w"> </span><span class="o">&&</span><span class="w"> </span>mv<span class="w"> </span>gswe-enumtypes.c.tmp<span class="w"> </span>gswe-enumtypes.c
</pre></div>
<p><tt class="docutils literal">$(GLIB_MKENUMS)</tt> is set in <tt class="docutils literal">configure</tt> using</p>
<div class="highlight"><pre><span></span>AC_PATH_PROG([GLIB_MKENUMS], [glib-mkenums])
</pre></div>
<p>This approach requires the <span class="caps">GNU</span> Autotools (you can get rid of it by changing <tt class="docutils literal">$(GLIB_MKENUMS)</tt> to
the path to <tt class="docutils literal"><span class="pre">glib-mkenums</span></tt> binary), and two template files, one for the header and one for the
code. <tt class="docutils literal">$(gswe_enum_headers)</tt> contains a list of all the header files that have enum types
defined throughout the project.</p>
<div class="highlight"><pre><span></span><span class="cm">/*** BEGIN file-header ***/</span>
<span class="cm">/* gswe-enumtypes.h - Enumeration types for SWE-GLib</span>
<span class="cm"> *</span>
<span class="cm"> * Copyright © 2013 Gergely Polonkai</span>
<span class="cm"> *</span>
<span class="cm"> * SWE-GLib is free software: you can redistribute it and/or modify</span>
<span class="cm"> * it under the terms of the GNU General Public License as published by</span>
<span class="cm"> * the Free Software Foundation; either version 3 of the License, or</span>
<span class="cm"> * (at your option) any later version.</span>
<span class="cm"> *</span>
<span class="cm"> * SWE-GLib is distributed in the hope that it will be useful,</span>
<span class="cm"> * but WITHOUT ANY WARRANTY; without even the implied warranty of</span>
<span class="cm"> * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the</span>
<span class="cm"> * GNU General Public License for more details.</span>
<span class="cm"> *</span>
<span class="cm"> * You should have received a copy of the GNU General Public License</span>
<span class="cm"> * along with this library; if not, see <http://www.gnu.org/licenses/>.</span>
<span class="cm"> */</span>
<span class="cp">#ifndef __GSWE_ENUM_TYPES_H__</span>
<span class="cp">#define __GSWE_ENUM_TYPES_H__</span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><glib-object.h></span>
<span class="cm">/*** END file-header ***/</span>
<span class="cm">/*** BEGIN file-production ***/</span>
<span class="cm">/* enumerations from "@filename@" */</span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf">"@filename@"</span>
<span class="cm">/*** END file-production ***/</span>
<span class="cm">/*** BEGIN value-header ***/</span>
<span class="n">GType</span><span class="w"> </span><span class="err">@</span><span class="n">enum_name</span><span class="err">@</span><span class="n">_get_type</span><span class="p">(</span><span class="kt">void</span><span class="p">);</span>
<span class="cp">#define @ENUMPREFIX@_TYPE_@ENUMSHORT@ (@enum_name@_get_type())</span>
<span class="cm">/*** END value-header ***/</span>
<span class="cm">/*** BEGIN file-tail ***/</span>
<span class="cp">#endif </span><span class="cm">/* __GSWE_ENUM_TYPES_H__ */</span>
<span class="cm">/*** END file-tail ***/</span>
</pre></div>
<div class="highlight"><pre><span></span><span class="cm">/*** BEGIN file-header ***/</span>
<span class="cm">/* gswe-enumtypes.c - Enumeration types for SWE-GLib</span>
<span class="cm"> *</span>
<span class="cm"> * Copyright © 2013 Gergely Polonkai</span>
<span class="cm"> *</span>
<span class="cm"> * SWE-GLib is free software: you can redistribute it and/or modify</span>
<span class="cm"> * it under the terms of the GNU General Public License as published by</span>
<span class="cm"> * the Free Software Foundation; either version 3 of the License, or</span>
<span class="cm"> * (at your option) any later version.</span>
<span class="cm"> *</span>
<span class="cm"> * SWE-GLib is distributed in the hope that it will be useful,</span>
<span class="cm"> * but WITHOUT ANY WARRANTY; without even the implied warranty of</span>
<span class="cm"> * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the</span>
<span class="cm"> * GNU General Public License for more details.</span>
<span class="cm"> *</span>
<span class="cm"> * You should have received a copy of the GNU General Public License</span>
<span class="cm"> * along with this library; if not, see <http://www.gnu.org/licenses/>.</span>
<span class="cm">*/</span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf">"swe-glib.h"</span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf">"gswe-enumtypes.h"</span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf">"@filename@"</span>
<span class="cm">/*** END file-header ***/</span>
<span class="cm">/*** BEGIN file-production ***/</span>
<span class="cm">/* enumerations from "@filename@" */</span>
<span class="cm">/*** END file-production ***/</span>
<span class="cm">/*** BEGIN value-header ***/</span>
<span class="n">GType</span>
<span class="err">@</span><span class="n">enum_name</span><span class="err">@</span><span class="n">_get_type</span><span class="p">(</span><span class="kt">void</span><span class="p">)</span>
<span class="p">{</span>
<span class="w"> </span><span class="k">static</span><span class="w"> </span><span class="k">volatile</span><span class="w"> </span><span class="n">gsize</span><span class="w"> </span><span class="n">g_define_type_id__volatile</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span>
<span class="w"> </span><span class="n">gswe_init</span><span class="p">();</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">g_once_init_enter</span><span class="p">(</span><span class="o">&</span><span class="n">g</span><span class="p">;</span><span class="n">_define_type_id__volatile</span><span class="p">))</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">static</span><span class="w"> </span><span class="k">const</span><span class="w"> </span><span class="n">G</span><span class="err">@</span><span class="n">Type</span><span class="err">@</span><span class="n">Value</span><span class="w"> </span><span class="n">values</span><span class="p">[]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{</span>
<span class="cm">/*** END value-header ***/</span>
<span class="cm">/*** BEGIN value-production ***/</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="err">@</span><span class="n">VALUENAME</span><span class="err">@</span><span class="p">,</span>
<span class="w"> </span><span class="s">"@VALUENAME@"</span><span class="p">,</span>
<span class="w"> </span><span class="s">"@valuenick@"</span>
<span class="w"> </span><span class="p">},</span>
<span class="cm">/*** END value-production ***/</span>
<span class="cm">/*** BEGIN value-tail ***/</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="nb">NULL</span><span class="p">,</span><span class="w"> </span><span class="nb">NULL</span><span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">};</span>
<span class="w"> </span><span class="n">GType</span><span class="w"> </span><span class="n">g_define_type_id</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">g_</span><span class="err">@</span><span class="n">type</span><span class="err">@</span><span class="n">_register_static</span><span class="p">(</span>
<span class="w"> </span><span class="n">g_intern_static_string</span><span class="p">(</span><span class="s">"@EnumName@"</span><span class="p">),</span>
<span class="w"> </span><span class="n">values</span>
<span class="w"> </span><span class="p">);</span>
<span class="w"> </span><span class="n">g_once_init_leave</span><span class="p">(</span><span class="o">&</span><span class="n">g</span><span class="p">;</span><span class="n">_define_type_id__volatile</span><span class="p">,</span><span class="w"> </span><span class="n">g_define_type_id</span><span class="p">);</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">g_define_type_id__volatile</span><span class="p">;</span>
<span class="p">}</span>
<span class="cm">/*** END value-tail ***/</span>
</pre></div>
List Git branches and their remote tracking branches side by side2014-07-18T21:46:45+00:002014-07-18T21:46:45+00:00Gergely Polonkaitag:None,2014-07-18:blog/2014/7/18/list-git-branches-and-their-remote-tracking-branches-side-by-side.html<p>I had a hard time following my own branches in a project. They got pretty numerous, and I wasn’t
sure if I pushed them to <tt class="docutils literal">origin</tt> at all. <tt class="docutils literal">git branch <span class="pre">-a</span></tt> can list all the branches,
including remote ones, but, as my list grew too big, it was impossible …</p><p>I had a hard time following my own branches in a project. They got pretty numerous, and I wasn’t
sure if I pushed them to <tt class="docutils literal">origin</tt> at all. <tt class="docutils literal">git branch <span class="pre">-a</span></tt> can list all the branches,
including remote ones, but, as my list grew too big, it was impossible to follow it any more.</p>
<p>Thus, I have created a small script called <tt class="docutils literal"><span class="pre">git-branches-with-remotes</span></tt>, which does the work for
me. Its only requirements are git (of course), and the <tt class="docutils literal">column</tt> command, which is pretty
obviously present on every <span class="caps">POSIX</span> compliant systems (even <span class="caps">OSX</span>).</p>
<div class="highlight"><pre><span></span><span class="ch">#! /bin/sh</span>
<span class="nv">COLUMN</span><span class="o">=</span><span class="sb">`</span>which<span class="w"> </span>column<span class="w"> </span><span class="m">2</span>><span class="w"> </span>/dev/null<span class="sb">`</span>
<span class="k">if</span><span class="w"> </span><span class="nb">test</span><span class="w"> </span>-z<span class="w"> </span><span class="nv">$COLUMN</span>
<span class="k">then</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"\`column' is not found in PATH. Cannot continue."</span>
<span class="w"> </span><span class="nb">exit</span><span class="w"> </span><span class="m">1</span>
<span class="k">fi</span>
<span class="nv">current_branch</span><span class="o">=</span><span class="sb">`</span>git<span class="w"> </span>rev-parse<span class="w"> </span>--abbrev-ref<span class="w"> </span>HEAD<span class="sb">`</span>
<span class="k">for</span><span class="w"> </span>branch<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="k">$(</span>git<span class="w"> </span><span class="k">for</span>-each-ref<span class="w"> </span>--shell<span class="w"> </span>--format<span class="o">=</span><span class="s1">'%(refname)'</span><span class="w"> </span>refs/heads<span class="w"> </span><span class="p">|</span><span class="w"> </span>sed<span class="w"> </span>-e<span class="w"> </span>s/^<span class="se">\'</span>refs<span class="se">\\</span>/heads<span class="se">\\</span>///<span class="w"> </span>-e<span class="w"> </span>s/<span class="se">\'</span>$//<span class="k">)</span>
<span class="k">do</span>
<span class="w"> </span><span class="nv">remote</span><span class="o">=</span><span class="sb">`</span>git<span class="w"> </span>config<span class="w"> </span>branch.<span class="nv">$branch</span>.remote<span class="sb">`</span>
<span class="w"> </span><span class="nv">merge</span><span class="o">=</span><span class="sb">`</span>git<span class="w"> </span>config<span class="w"> </span>branch.<span class="nv">$branch</span>.merge<span class="w"> </span><span class="p">|</span><span class="w"> </span>sed<span class="w"> </span>-e<span class="w"> </span><span class="s1">'s/^refs\/heads\///'</span><span class="sb">`</span>
<span class="w"> </span><span class="o">[</span><span class="w"> </span>x<span class="s2">"</span><span class="nv">$current_branch</span><span class="s2">"</span><span class="w"> </span><span class="o">==</span><span class="w"> </span>x<span class="s2">"</span><span class="nv">$branch</span><span class="s2">"</span><span class="w"> </span><span class="o">]</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="nb">echo</span><span class="w"> </span>-n<span class="w"> </span><span class="s1">'*'</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span>-n<span class="w"> </span><span class="s2">"</span><span class="nv">$branch</span><span class="s2">"</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span>!<span class="w"> </span><span class="nb">test</span><span class="w"> </span>-z<span class="w"> </span><span class="nv">$merge</span>
<span class="w"> </span><span class="k">then</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span>-en<span class="w"> </span><span class="s2">"\t"</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span>-n<span class="w"> </span><span class="nv">$remote</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span>-n<span class="w"> </span>/
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span>-n<span class="w"> </span><span class="nv">$merge</span>
<span class="w"> </span><span class="k">fi</span>
<span class="w"> </span><span class="nb">echo</span>
<span class="k">done</span><span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="nv">$COLUMN</span><span class="w"> </span>-t
</pre></div>
<p>I just put it in my path, and <tt class="docutils literal">git <span class="pre">branches-with-remotes</span></tt> does the work!</p>
<p>Edit (16 August): I have added some code to mark the current branch (if any) with an asterisk.
Also, I have put this script <a class="reference external" href="https://gist.github.com/gergelypolonkai/8af6a3e86b57dd4c250e">in a gist</a>.</p>
<p>Edit (26 February, 2015): It turns out that <tt class="docutils literal">git branch <span class="pre">-vv</span></tt> shows the same information and some
more: it also shows if the branches are diverged, and the first line of the last commit’s message.</p>
First impressions of Windows 82013-11-05T08:14:50+00:002013-11-05T08:14:50+00:00Gergely Polonkaitag:None,2013-11-05:blog/2013/11/5/first-impressions-of-windows-8.html<p>Many of you may know my commitment to Linux and Open Source Software. But this doesn’t mean I
hate proprietary software like many others do. I think everything has its own place in the world,
and this goes for software as well.</p>
<p>A few days ago I got my …</p><p>Many of you may know my commitment to Linux and Open Source Software. But this doesn’t mean I
hate proprietary software like many others do. I think everything has its own place in the world,
and this goes for software as well.</p>
<p>A few days ago I got my hands on a new notebook, thanks to my company. It was shipped with
Windows 8 by default, and although I installed Fedora 19 in an instant (which went smoothlessly,
even with Secure Boot enabled), I’ve decided to give a try to this new Windows Version.</p>
<p>Being a heavy Windows 7 user, my first thought was “What the hell is this?” But in a day, I got
totally used to it. I don’t miss the Start button at all. The applications already installed
were almost enough for me (I still need Office. Maybe I’ll also enroll for Office 365 later…),
and the games are great and beautiful too. So overall, this new version may be totally different
(by the looks), but it seems almost the same Windows as we know it. So if you don’t freak out by
touching something new, go give it a try: don’t instant-remove 8 in favour of 7!</p>
From Symfony to Django in two days2013-09-24T14:05:22+00:002013-09-24T14:05:22+00:00Gergely Polonkaitag:None,2013-09-24:blog/2013/9/24/from-symfony-to-django-in-two-days.html<p>I was a Python hater for a long time, although I can’t really tell why. It didn’t fit in my mind,
maybe. I was programming in <span class="caps">BASIC</span>, Pascal (none of these would come to my mind now, though), C,
<span class="caps">PHP</span>, Perl, JavaScript, and different shell “languages” like awk …</p><p>I was a Python hater for a long time, although I can’t really tell why. It didn’t fit in my mind,
maybe. I was programming in <span class="caps">BASIC</span>, Pascal (none of these would come to my mind now, though), C,
<span class="caps">PHP</span>, Perl, JavaScript, and different shell “languages” like awk, sed or bash.</p>
<p>After I could not fit my next Symfony app on my cloud server (it is pretty low on storage), I have
decided to move slowly to Django. My first task was simple: transition my web page (this one)
from <span class="caps">PHP</span> + Symfony 2 to Python + Django. The results: the “static” pages are already working, the
blog listing is almost ready (some styling issues are still around), only tagging remains. And
this is after about 6 hours of work. Oh, and the admin site is included with Django, so I don’t
have to port that. I have also decided to finally integrate a comment feature in the Django version.</p>
SWE-GLib final release2013-09-16T21:37:17+00:002013-09-16T21:37:17+00:00Gergely Polonkaitag:None,2013-09-16:blog/2013/9/16/swe-glib-final-release.html<p>Few of you may know that I’m interested in astrology. About two months ago I have decided to
create an astrologers’ software for the <span class="caps">GNOME</span> desktop. Since then, I have contacted Jean-André
Santoni, who created a software called <a class="reference external" href="https://code.google.com/p/astrognome/">Astrognome</a> some
years ago. We exchanged some e-mails, and after several …</p><p>Few of you may know that I’m interested in astrology. About two months ago I have decided to
create an astrologers’ software for the <span class="caps">GNOME</span> desktop. Since then, I have contacted Jean-André
Santoni, who created a software called <a class="reference external" href="https://code.google.com/p/astrognome/">Astrognome</a> some
years ago. We exchanged some e-mails, and after several weeks of coding, I’m proud to present
<a class="reference external" href="https://github.com/gergelypolonkai/swe-glib"><span class="caps">SWE</span>-GLib</a> 1.0.1. This is “just” a library which
wraps <a class="reference external" href="http://www.astro.com/swisseph/">Swiss Ephemeris</a>, creating a nice GLib-ish interface
around it. See the project page and the built-in <span class="caps">GTK</span>-Doc document for more information.</p>
<p>The astrologer’s software I’m writing will be Astrognome (you can check the <a class="reference external" href="https://github.com/gergelypolonkai/astrognome">GitHub repository</a> already, thanks for Jean-André for letting me
use the name). It is currently in pre-alpha status, but already utilizes <span class="caps">SWE</span>-GLib (it just can’t
display the results yet). If you happen to be interested in astrology and/or Astrognome, fork the
repository and contribute! You can also contact me (or open an enhancement issue on GitHub) if
you have any ideas.</p>
Installing OTRS in Fedora 18 with SELinux enabled2013-05-06T06:01:52+00:002013-05-06T06:01:52+00:00Gergely Polonkaitag:None,2013-05-06:blog/2013/5/6/installing-otrs-in-fedora-18-with-selinux-enabled.html<p>I’ve read somewhere in an <span class="caps">OTRS</span> installation howto that if you want to install <span class="caps">OTRS</span>, you will have
to disable SELinux. Well, I won’t.</p>
<p>During the last few months, I have been using Fedora 18 with SELinux on all of my desktop machines
and on my notebook, and …</p><p>I’ve read somewhere in an <span class="caps">OTRS</span> installation howto that if you want to install <span class="caps">OTRS</span>, you will have
to disable SELinux. Well, I won’t.</p>
<p>During the last few months, I have been using Fedora 18 with SELinux on all of my desktop machines
and on my notebook, and I had no problems at all. Meanwhile I got familiar with SELinux itself,
and got used to solving problems caused by it. So I started <tt class="docutils literal">tail <span class="pre">-f</span> /var/log/httpd/error_log</tt>
in one terminal (to see if something Apache related thing appears), <tt class="docutils literal">tail <span class="pre">-f</span>
/var/log/audit/audit.log</tt> in another (to see errors caused by SELinux), opened the admin manual
at the installation chapter, took a deep breath, and went on.</p>
<p>Throughout this article, I will refer to <span class="caps">OTRS</span> 3.2.6 as <span class="caps">OTRS</span> and Fedora 18 (with only “stock”
repositories) as Fedora. I assume that you have already installed <span class="caps">OTRS</span> in a non-SELinux
environment before, and that you have at least some basic knowledge about SELinux, <span class="caps">MAC</span>, <span class="caps">RBAC</span>, and
all the like. I’m installing <span class="caps">OTRS</span> in <tt class="docutils literal">/opt/otrs</tt>, so if you install it somewhere else, you will
have to modify the paths below. Also, if you happen to install under <tt class="docutils literal">/var/www</tt> (I wouldn’t
recommend it), that directory already has the <tt class="docutils literal">httpd_sys_content_t</tt> type, so you won’t have to
set it explicitly.</p>
<p>As the first step I have unpacked the archive to <tt class="docutils literal">/opt/otrs</tt>. This directory is the <span class="caps">OTRS</span>
default, many config files have it hardcoded, and changing it is no easy task.</p>
<p>Running <tt class="docutils literal">otrs.CheckModules.pl</tt> gave me a list of missing perl modules. Red Hat and Fedora makes
it easy to install these, as you don’t have to know the <span class="caps">RPM</span> package name, just the perl module name:</p>
<div class="highlight"><pre><span></span>yum<span class="w"> </span>install<span class="w"> </span><span class="s1">'perl(Crypt::SSLeay)'</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="s1">'perl(DBD::Pg)'</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="s1">'perl(GD)'</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="s1">'perl(JSON::XS)'</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="s1">'perl(GD::Text)'</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="s1">'perl(GD::Graph)'</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="s1">'perl(Mail::IMAPClient)'</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="s1">'perl(Net::DNS)'</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="s1">'perl(PDF::API2)'</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="s1">'perl(Text::CSV_XS)'</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="s1">'perl(YAML::XS)'</span>
</pre></div>
<p>I also needed to install <tt class="docutils literal">mod_perl</tt>. Although <tt class="docutils literal">otrs.CheckModules.pl</tt> didn’t mention it, the
default settings use syslog as the logging module, so unless you change it in <tt class="docutils literal">Config.pm</tt>, you
will also need to install <tt class="docutils literal"><span class="pre">'perl(Unix::Syslog)'</span></tt>, either.</p>
<p>The default SELinux policy doesn’t permit any network connection to be initiated by Apache httpd.
As <span class="caps">OTRS</span> needs to connect to its database, you need to enable it explicitly. In older
distributions, the <tt class="docutils literal">httpd_can_network_connect</tt> was the SELinux boolean for this, but recent
installations also have a <tt class="docutils literal">httpd_can_network_connect_db</tt> flag. As far as I know, this enables
all network connections to the well-known database servers’ default port, but I will have to check
for it. For me, with a MySQL listening on its standard port, the <tt class="docutils literal">setsebool
httpd_can_network_connect_db=1</tt> command just did it.</p>
<p>With SELinux enabled, Apache won’t be able to read anything that’s not marked with the
<tt class="docutils literal">httpd_sys_content_t</tt> type, nor write anywhere without the <tt class="docutils literal">httpd_sys_rw_content_t</tt> type. The
trivial, quick and dirty solution is to label all the files as <tt class="docutils literal">httpd_sys_rw_content_t</tt>, and let
everything go. However, the goal of SELinux is just the opposite of this: grant access only to
what is really needed. After many trial-and-error steps, it finally turned out that for <span class="caps">OTRS</span> to
work correctly, you must set</p>
<ul class="simple">
<li><tt class="docutils literal">httpd_sys_content_t</tt><ul>
<li>on <tt class="docutils literal">/opt/otrs/var/httpd/htdocs</tt></li>
</ul>
</li>
<li><tt class="docutils literal">httpd_script_exec_t</tt><ul>
<li>on <tt class="docutils literal"><span class="pre">/opt/otrs/bin/cgi-bin</span></tt></li>
</ul>
</li>
<li><tt class="docutils literal">httpd_sys_rw_content_t</tt><ul>
<li>on <tt class="docutils literal">/opt/otrs/Kernel</tt></li>
<li>on <tt class="docutils literal">/opt/otrs/var/sessions</tt></li>
<li>on <tt class="docutils literal">/opt/otrs/var/log</tt> (unless you use syslog for logging)</li>
<li>on <tt class="docutils literal">/opt/otrs/var/packages</tt> (this is used only when you download an .opm package)</li>
<li>on <tt class="docutils literal">/opt/otrs/var/stats</tt></li>
<li>on <tt class="docutils literal">/opt/otrs/var/tmp</tt></li>
<li>on <tt class="docutils literal">/opt/otrs/bin</tt> (I wonder why this is required, though)</li>
</ul>
</li>
</ul>
<p>To do this, use the following command:</p>
<div class="highlight"><pre><span></span>semanage<span class="w"> </span>fcontext<span class="w"> </span>-a<span class="w"> </span>-t<span class="w"> </span><context><span class="w"> </span><directory<span class="w"> </span>regex>
</pre></div>
<p>Where <tt class="docutils literal"><directory regex></tt> is something like <tt class="docutils literal"><span class="pre">/opt/otrs/Kernel(/.*)?</span></tt>. When this is done, all
you have to do is running <tt class="docutils literal">restorecon <span class="pre">-vR</span> /opt/otrs</tt> so it will relabel everything with the
correct types (you can omit <tt class="docutils literal"><span class="pre">-v</span></tt>, I just like to see what my software do).</p>
<p>The last thing I faced is that Fedora is more restrictive on reading directories other than
<tt class="docutils literal">/var/www</tt>. It has a <tt class="docutils literal">Require all denied</tt> on <tt class="docutils literal"><Directory /></tt>, and a <tt class="docutils literal">Require all granted</tt>
on <tt class="docutils literal"><Directory /var/www></tt>, so <tt class="docutils literal">/opt/otrs/var/httpd/htdocs</tt> will throw a <tt class="docutils literal">403 Forbidden
(client denied by server configuration)</tt> error. To get rid of this, I had to modify
<tt class="docutils literal"><span class="pre">scripts/apache2-httpd.include.conf</span></tt> and add <tt class="docutils literal">Require all granted</tt> to both the <tt class="docutils literal"><span class="pre">cgi-bin</span></tt> and
<tt class="docutils literal">htdocs</tt> directories.</p>
<p>As I will have to use <span class="caps">OTRS</span> in a production environment soon with SELinux enabled, it is more than
sure that this list will change in the near future. As there are no official documentation on
this (I haven’t find one yet), I have to do it with the trial-and-error way, so be patient!</p>
Renaming a Symfony 2 bundle2013-04-09T22:29:48+00:002013-04-09T22:29:48+00:00Gergely Polonkaitag:None,2013-04-09:blog/2013/4/9/renaming-a-symfony-2-bundle.html<p>Today I’ve realised that the name I gave to one of my Symfony 2 bundles should be something else.
To rename a bundle, one must do four things (at least).</p>
<ol class="arabic simple">
<li>Change the namespace from <tt class="docutils literal">Vendor\OldBundle</tt> to <tt class="docutils literal">Vendor\NewBundle</tt> in every <span class="caps">PHP</span> class
(sounds like pain? It is…)</li>
<li>Change …</li></ol><p>Today I’ve realised that the name I gave to one of my Symfony 2 bundles should be something else.
To rename a bundle, one must do four things (at least).</p>
<ol class="arabic simple">
<li>Change the namespace from <tt class="docutils literal">Vendor\OldBundle</tt> to <tt class="docutils literal">Vendor\NewBundle</tt> in every <span class="caps">PHP</span> class
(sounds like pain? It is…)</li>
<li>Change the name of files and classes. Some files under <tt class="docutils literal">src/Vendor/OldBundle</tt> (and the
classes in them) contain the name of the bundle, like
<tt class="docutils literal">OldBundle/DependencyInjection/VendorOldBundleExtension.php</tt> and
<tt class="docutils literal">OldBundle/VendorOldBundle.php</tt>. You should rename them, or Symfony won’t find the classes
defined in them! When done, rename the whole bundle directory either.</li>
<li>Change the configuration files accordingly, including <tt class="docutils literal">AppKernel.php</tt>. These config files
are usually <tt class="docutils literal">routing.yml</tt>, <tt class="docutils literal">services.yml</tt>, and in some cases, <tt class="docutils literal">config.yml</tt>.</li>
<li>Change the references in other parts of your code. A <tt class="docutils literal">grep <span class="pre">-r</span> OldBundle .</tt> will usually help…</li>
</ol>
Dvorak and me2013-03-13T21:20:13+00:002013-03-13T21:20:13+00:00Gergely Polonkaitag:None,2013-03-13:blog/2013/3/13/dvorak-and-me.html<p>A few months ago I have decided to switch to the Dvorak layout. After using <span class="caps">QWERTY</span> (well, <span class="caps">QWERTZ</span>,
to be precise) for almost 17 years, it was a hard decision, but now I think it worthed the try. I
started with the <span class="caps">UK</span> (Dvorak with <span class="caps">UK</span> punctuation) layout, and in …</p><p>A few months ago I have decided to switch to the Dvorak layout. After using <span class="caps">QWERTY</span> (well, <span class="caps">QWERTZ</span>,
to be precise) for almost 17 years, it was a hard decision, but now I think it worthed the try. I
started with the <span class="caps">UK</span> (Dvorak with <span class="caps">UK</span> punctuation) layout, and in about four weeks, I’ve almost
reached my original typing speed. Today I have modified the Hungarian xkb definitions file to add
the Hungarian accended letters like ű to the layout, so I don’t have to use dead keys anymore
(which apparently turned out to be a problem, as the Linux version of Java doesn’t support dead
keys at all).</p>
<p>Best thing is, as I never learned proper 10-finger typing, but learned Dvorak that way, I can
switch between <span class="caps">QWERTY</span> and Dvorak more or less painlessly (about 10 minutes of confusion, so to say).</p>
<p>Conclusion: I don’t know yet if this was actually a good decision, but it wasn’t bad, after all.
But seeing people’s faces when they try to type on my machine totally worths it.</p>
Haversine in MySQL2013-03-05T12:49:28+00:002013-03-05T12:49:28+00:00Gergely Polonkaitag:None,2013-03-05:blog/2013/3/5/haversine-in-mysql.html<p>Just insert it in your database, feed them two Google coordinates, and you get the distance in
kilometres. If you happen to need it in miles, change the constant <tt class="docutils literal">12756.200</tt> in the
<tt class="docutils literal"><span class="caps">RETURN</span></tt> row to <tt class="docutils literal">7922.6</tt> instead.</p>
<div class="highlight"><pre><span></span><span class="k">DELIMITER</span><span class="w"> </span><span class="err">$$</span>
<span class="k">CREATE</span><span class="w"> </span><span class="k">FUNCTION</span><span class="w"> </span><span class="o">`</span><span class="n">haversine</span><span class="o">`</span><span class="w"> </span><span class="p">(</span><span class="n">lng1</span><span class="w"> </span><span class="nb">FLOAT</span><span class="p">,</span><span class="w"> </span><span class="n">lat1</span><span class="w"> </span><span class="nb">FLOAT</span><span class="p">,</span><span class="w"> </span><span class="n">lng2</span><span class="w"> </span><span class="nb">FLOAT</span><span class="p">,</span><span class="w"> </span><span class="n">lat2 …</span></pre></div><p>Just insert it in your database, feed them two Google coordinates, and you get the distance in
kilometres. If you happen to need it in miles, change the constant <tt class="docutils literal">12756.200</tt> in the
<tt class="docutils literal"><span class="caps">RETURN</span></tt> row to <tt class="docutils literal">7922.6</tt> instead.</p>
<div class="highlight"><pre><span></span><span class="k">DELIMITER</span><span class="w"> </span><span class="err">$$</span>
<span class="k">CREATE</span><span class="w"> </span><span class="k">FUNCTION</span><span class="w"> </span><span class="o">`</span><span class="n">haversine</span><span class="o">`</span><span class="w"> </span><span class="p">(</span><span class="n">lng1</span><span class="w"> </span><span class="nb">FLOAT</span><span class="p">,</span><span class="w"> </span><span class="n">lat1</span><span class="w"> </span><span class="nb">FLOAT</span><span class="p">,</span><span class="w"> </span><span class="n">lng2</span><span class="w"> </span><span class="nb">FLOAT</span><span class="p">,</span><span class="w"> </span><span class="n">lat2</span><span class="w"> </span><span class="nb">FLOAT</span><span class="p">)</span>
<span class="k">RETURNS</span><span class="w"> </span><span class="nb">float</span><span class="w"> </span><span class="k">NO</span><span class="w"> </span><span class="k">SQL</span><span class="w"> </span><span class="k">DETERMINISTIC</span>
<span class="k">BEGIN</span>
<span class="w"> </span><span class="k">SET</span><span class="w"> </span><span class="o">@</span><span class="n">a</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">ABS</span><span class="p">(</span><span class="n">POWER</span><span class="p">(</span><span class="n">SIN</span><span class="p">(</span><span class="n">RADIANS</span><span class="p">(</span><span class="n">lat1</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">lat2</span><span class="p">))</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="mi">2</span><span class="p">,</span><span class="w"> </span><span class="mi">2</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">COS</span><span class="p">(</span><span class="n">RADIANS</span><span class="p">(</span><span class="n">lat1</span><span class="p">))</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">COS</span><span class="p">(</span><span class="n">RADIANS</span><span class="p">(</span><span class="n">lat2</span><span class="p">))</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">POWER</span><span class="p">(</span><span class="n">SIN</span><span class="p">(</span><span class="n">RADIANS</span><span class="p">(</span><span class="n">lng1</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">lng2</span><span class="p">))</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="mi">2</span><span class="p">,</span><span class="w"> </span><span class="mi">2</span><span class="p">));</span>
<span class="w"> </span><span class="k">RETURN</span><span class="w"> </span><span class="mi">12756</span><span class="p">.</span><span class="mi">200</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">ATAN2</span><span class="p">(</span><span class="n">SQRT</span><span class="p">(</span><span class="o">@</span><span class="n">a</span><span class="p">),</span><span class="w"> </span><span class="n">SQRT</span><span class="p">(</span><span class="mi">1</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="o">@</span><span class="n">a</span><span class="p">));</span>
<span class="k">END</span><span class="err">$$</span>
</pre></div>
Fedora can’t change Active Directory password via kpasswd2013-03-05T08:55:04+00:002013-03-05T08:55:04+00:00Gergely Polonkaitag:None,2013-03-05:blog/2013/3/5/fedora-can-t-change-active-directory-password-via-kpasswd.html<p>I wanted to change my <span class="caps">AD</span> password today. As the <span class="caps">AD</span> is actually a Kerberos server, I was pretty
sure that <tt class="docutils literal">kpasswd</tt> will do the trick. However, <tt class="docutils literal">kpasswd</tt> output looked like this:</p>
<div class="highlight"><pre><span></span><span class="go">$ kpasswd</span>
<span class="go">Password for polonkai.gergely@EXAMPLE.LOCAL:</span>
<span class="go">Enter new password:</span>
<span class="go">Enter it again:</span>
<span class="go">kpasswd: Cannot find KDC for …</span></pre></div><p>I wanted to change my <span class="caps">AD</span> password today. As the <span class="caps">AD</span> is actually a Kerberos server, I was pretty
sure that <tt class="docutils literal">kpasswd</tt> will do the trick. However, <tt class="docutils literal">kpasswd</tt> output looked like this:</p>
<div class="highlight"><pre><span></span><span class="go">$ kpasswd</span>
<span class="go">Password for polonkai.gergely@EXAMPLE.LOCAL:</span>
<span class="go">Enter new password:</span>
<span class="go">Enter it again:</span>
<span class="go">kpasswd: Cannot find KDC for requested realm changing password</span>
</pre></div>
<p>I’ve checked <tt class="docutils literal">kinit</tt> and <tt class="docutils literal">klist</tt>, everything looked fine. After a while it came
to my mind that password changing is done through the kadmin server, not
through the <span class="caps">KDC</span>. It seems that when I set up the Active Directory membership,
the <tt class="docutils literal">admin_server</tt> directive is not get written to <tt class="docutils literal">krb5.conf</tt>. So all I had to
do was to put</p>
<div class="highlight"><pre><span></span>admin_server = ad.example.local
</pre></div>
<p>in that file, and voilà!</p>
<div class="highlight"><pre><span></span><span class="go">$ kpasswd</span>
<span class="go">Password for polonkai.gergely@EXAMPLE.LOCAL:</span>
<span class="go">Enter new password:</span>
<span class="go">Enter it again:</span>
<span class="go">Password changed.</span>
</pre></div>
Why I stopped using annotation based routing in Symfony today2013-02-27T23:10:24+00:002013-02-27T23:10:24+00:00Gergely Polonkaitag:None,2013-02-27:blog/2013/2/27/why-i-stopped-using-annotation-based-routing-in-symfony-today.html<p>I have read several opinions about routing configuration in Symfony. I stayed with annotation
based routing as it was convinient for me to see the <span class="caps">URL</span> right above the controller action. This
was because by just checking the <span class="caps">URL</span>, I remembered the controlling code, as they always were fresh
ones …</p><p>I have read several opinions about routing configuration in Symfony. I stayed with annotation
based routing as it was convinient for me to see the <span class="caps">URL</span> right above the controller action. This
was because by just checking the <span class="caps">URL</span>, I remembered the controlling code, as they always were fresh
ones. Well, until today.</p>
<p>I had to take a look into an old (Sf 2.0, last commit was about 3 months ago) project of mine. In
the same run I’ve upgraded the whole project to 2.2 (it was a fast one, thanks for <a class="reference external" href="https://github.com/jmikola">JMikola@GitHub</a> for the quick reply on my issue with <a class="reference external" href="https://github.com/jmikola/JmikolaJsAssetsHelperBundle">JmikolaJsAssetsHelperBundle</a> again!). After that I went on to the
requested change. Now, finding a route in about 40 controller files spread between 3 bundles can
really be a pain! So I’ve finished with annotation based routing. It’s still a nice feature,
it’s simply not for me.</p>
mount: device or resource busy after enabling multipath2013-02-19T23:09:05+00:002013-02-19T23:09:05+00:00Gergely Polonkaitag:None,2013-02-19:blog/2013/2/19/mount-device-or-resource-busy-after-enabling-multipath.html<p>We have a heartbeat cluster with two nodes. It has been running for several months without
problems. The shared storage is on an <span class="caps">IBM</span> <span class="caps">DS3400</span>, on which we have a large volume formatted with ext4.</p>
<p>Today I decided to reboot the active node for security reasons. So I’ve switched …</p><p>We have a heartbeat cluster with two nodes. It has been running for several months without
problems. The shared storage is on an <span class="caps">IBM</span> <span class="caps">DS3400</span>, on which we have a large volume formatted with ext4.</p>
<p>Today I decided to reboot the active node for security reasons. So I’ve switched to the passive
node, which failed at the first step: it was unable to mount the storage (<tt class="docutils literal">/dev/sda1</tt>). After
whining for a few moments, I tried to mount it by hand, which told me</p>
<div class="highlight"><pre><span></span>/dev/sda1 already mounted or /data is busy
</pre></div>
<p>I’ve quickly made sure that none of that was true. After checking this-and-that, it turned out
that the passive node had <tt class="docutils literal">multipathd</tt> running, so I looked under <tt class="docutils literal">/dev/mapper</tt>, and found two
symlinks there, <tt class="docutils literal"><span class="pre"><long-long</span> <span class="caps">WWN</span>></tt> and <tt class="docutils literal"><span class="pre"><long-long</span> <span class="pre"><span class="caps">WWN</span>>-part1</span></tt>. As the partition table and the
disk size was the same as on <tt class="docutils literal">/dev/sda</tt>, I tried to</p>
<div class="highlight"><pre><span></span>mount<span class="w"> </span>/dev/<long-long<span class="w"> </span>WWN>-part1<span class="w"> </span>/data
</pre></div>
<p>and voilà! It worked like charm!</p>
JMS\DiExtraBundle’s GrepPatternFinder – grep exits with status code 2 on Fedora 182013-01-17T00:32:12+00:002013-01-17T00:32:12+00:00Gergely Polonkaitag:None,2013-01-17:blog/2013/1/17/jms-diextrabundle-s-greppatternfinder-grep-exits-with-status-code-2-on-fedora-18.html<p>Yesterday I’ve upgraded my development machines from Fedora 17 to Fedora
18. Although it went well, my <a class="reference external" href="http://symfony.com/">Symfony</a> projects stopped working with a
message like this:</p>
<div class="highlight"><pre><span></span>RuntimeException: Command "/usr/bin/grep --fixed-strings --directories=recurse --devices=skip --files-with-matches --with-filename --color=never --include=*.php 'JMS\DiExtraBundle\Annotation'
'/var/www/html/gergelypolonkaiweb …</pre></div><p>Yesterday I’ve upgraded my development machines from Fedora 17 to Fedora
18. Although it went well, my <a class="reference external" href="http://symfony.com/">Symfony</a> projects stopped working with a
message like this:</p>
<div class="highlight"><pre><span></span>RuntimeException: Command "/usr/bin/grep --fixed-strings --directories=recurse --devices=skip --files-with-matches --with-filename --color=never --include=*.php 'JMS\DiExtraBundle\Annotation'
'/var/www/html/gergelypolonkaiweb/app/../src'
'/var/www/html/gergelypolonkaiweb/vendor/symfony/symfony/src/Symfony/Bundle/FrameworkBundle'
'/var/www/html/gergelypolonkaiweb/vendor/symfony/symfony/src/Symfony/Bundle/SecurityBundle'
'/var/www/html/gergelypolonkaiweb/vendor/symfony/symfony/src/Symfony/Bundle/TwigBundle'
'/var/www/html/gergelypolonkaiweb/vendor/symfony/monolog-bundle/Symfony/Bundle/MonologBundle'
'/var/www/html/gergelypolonkaiweb/vendor/symfony/swiftmailer-bundle/Symfony/Bundle/SwiftmailerBundle'
'/var/www/html/gergelypolonkaiweb/vendor/symfony/assetic-bundle/Symfony/Bundle/AsseticBundle'
'/var/www/html/gergelypolonkaiweb/vendor/doctrine/doctrine-bundle/Doctrine/Bundle/DoctrineBundle'
'/var/www/html/gergelypolonkaiweb/vendor/sensio/framework-extra-bundle/Sensio/Bundle/FrameworkExtraBundle'
'/var/www/html/gergelypolonkaiweb/vendor/jms/aop-bundle/JMS/AopBundle'
'/var/www/html/gergelypolonkaiweb/vendor/jms/security-extra-bundle/JMS/SecurityExtraBundle'
'/var/www/html/gergelypolonkaiweb/vendor/doctrine/doctrine-migrations-bundle/Doctrine/Bundle/MigrationsBundle'
'/var/www/html/gergelypolonkaiweb/vendor/friendsofsymfony/jsrouting-bundle/FOS/JsRoutingBundle'
'/var/www/html/gergelypolonkaiweb/vendor/avalanche123/imagine-bundle/Avalanche/Bundle/ImagineBundle'
'/var/www/html/gergelypolonkaiweb/vendor/genemu/form-bundle/Genemu/Bundle/FormBundle'
'/var/www/html/gergelypolonkaiweb/src/GergelyPolonkai/FrontBundle'
'/var/www/html/gergelypolonkaiweb/src/GergelyPolonkai/GeshiBundle'
'/var/www/html/gergelypolonkaiweb/vendor/symfony/symfony/src/Symfony/Bundle/WebProfilerBundle'
'/var/www/html/gergelypolonkaiweb/vendor/sensio/distribution-bundle/Sensio/Bundle/DistributionBundle'
'/var/www/html/gergelypolonkaiweb/vendor/sensio/generator-bundle/Sensio/Bundle/GeneratorBundle'" exited with non-successful status code "2".
</pre></div>
<p>After getting through my logs and such, I’ve finally found out that the new SELinux policy is
causing the trouble together with git. Eventually, my <tt class="docutils literal">.git/logs</tt> directory is tagged as
<tt class="docutils literal">unconfined_u:object_r:httpd_log_t:s0</tt>. <tt class="docutils literal">httpd_log_t</tt> type is not readable by the
<tt class="docutils literal">system_u:system_r:httpd_t:s0</tt> user, which makes <tt class="docutils literal">/usr/bin/grep</tt> throw an access denied error.
To fix this, I needed to do</p>
<div class="highlight"><pre><span></span>semanage<span class="w"> </span>fcontext<span class="w"> </span>-a<span class="w"> </span>-t<span class="w"> </span>httpd_sys_content_t<span class="w"> </span><span class="s1">'/var/www(/.*)?/\.git/logs(/.*)?'</span>
</pre></div>
<p>as root. This makes <tt class="docutils literal">.git</tt> directories readable for the httpd process, thus, for <tt class="docutils literal">grep</tt>. The
optimal solution would be to tell <tt class="docutils literal">GrepPatternFinder</tt> to ignore version control stuff, so the
<tt class="docutils literal">httpd</tt> process would have no access to them at all. Also, in production, removing the <tt class="docutils literal">.git</tt>
or <tt class="docutils literal">.svn</tt> directories could be a good idea.</p>
git rm —cached madness2013-01-14T21:38:00+00:002013-01-14T21:38:00+00:00Gergely Polonkaitag:None,2013-01-14:blog/2013/1/14/git-rm-cached-madness.html<p>I have recently learned about <tt class="docutils literal">git rm <span class="pre">--cached</span></tt>. It’s a very good tool, as it removes a file
from tracking, without removing your local copy of it. However, be warned that if you use <tt class="docutils literal">git
pull</tt> in another working copy, the file will be removed from there! If you …</p><p>I have recently learned about <tt class="docutils literal">git rm <span class="pre">--cached</span></tt>. It’s a very good tool, as it removes a file
from tracking, without removing your local copy of it. However, be warned that if you use <tt class="docutils literal">git
pull</tt> in another working copy, the file will be removed from there! If you accidentally put the
configuration of a production project, and remove it on your dev machine, it can cause a lot of
trouble ;)</p>
Registering an enum type in GLib’s type system2013-01-06T02:34:03+00:002013-01-06T02:34:03+00:00Gergely Polonkaitag:None,2013-01-06:blog/2013/1/6/registering-an-enum-type-in-glib-s-type-system.html<p>I faced a problem in my <a class="reference external" href="https://developer.gnome.org/glib/">GLib</a> self-teaching project, <a class="reference external" href="https://github.com/gergelypolonkai/wmud">wMUD</a> today. I wanted to register a signal for a
<tt class="docutils literal">GObject</tt>, whose handler should accept two <tt class="docutils literal">enum</tt> parameters for which I had to register a new
<tt class="docutils literal">GEnum</tt> type in the <tt class="docutils literal">GObject</tt> type system. However, the <a class="reference external" href="https://developer.gnome.org/gobject/unstable/gtype-non-instantiable.html">documentation on this feature</a> (thanks for …</p><p>I faced a problem in my <a class="reference external" href="https://developer.gnome.org/glib/">GLib</a> self-teaching project, <a class="reference external" href="https://github.com/gergelypolonkai/wmud">wMUD</a> today. I wanted to register a signal for a
<tt class="docutils literal">GObject</tt>, whose handler should accept two <tt class="docutils literal">enum</tt> parameters for which I had to register a new
<tt class="docutils literal">GEnum</tt> type in the <tt class="docutils literal">GObject</tt> type system. However, the <a class="reference external" href="https://developer.gnome.org/gobject/unstable/gtype-non-instantiable.html">documentation on this feature</a> (thanks for pointing
out goes to <cite>hashem</cite> on <tt class="docutils literal"><span class="pre">#gnome-hackers</span></tt>) is not… uhm… obvious. Making the long story short, I
have checked with the <tt class="docutils literal"><span class="caps">GIO</span></tt> sources for an example, and using that, I have created this small,
working chunk:</p>
<div class="highlight"><pre><span></span><span class="cp">#ifndef __WMUD_CLIENT_STATE_H__</span>
<span class="cp">#define __WMUD_CLIENT_STATE_H__</span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><glib-object.h></span>
<span class="cm">/**</span>
<span class="cm"> * WmudClientState:</span>
<span class="cm"> * @WMUD_CLIENT_STATE_FRESH: Client is newly connected. Waiting for a login</span>
<span class="cm"> * player name</span>
<span class="cm"> * @WMUD_CLIENT_STATE_PASSWAIT: Login player name is entered, waiting for a</span>
<span class="cm"> * login password</span>
<span class="cm"> * @WMUD_CLIENT_STATE_MENU: Authentication was successful, player is now in the</span>
<span class="cm"> * main game menu</span>
<span class="cm"> * @WMUD_CLIENT_STATE_INGAME: Character login was successful, player is now</span>
<span class="cm"> * in-game</span>
<span class="cm"> * @WMUD_CLIENT_STATE_YESNO: Player was asked a yes/no question, and we are</span>
<span class="cm"> * waiting for the answer. client.yesNoCallback MUST be set at this point!</span>
<span class="cm"> * TODO: if wmudClient had a prevState field, and there would be some hooks</span>
<span class="cm"> * that are called before and after the client enters a new state, this</span>
<span class="cm"> * could be a three-state stuff, in which the player can enter e.g ? as</span>
<span class="cm"> * the answer, so they would be presented with the question again.</span>
<span class="cm"> * @WMUD_CLIENT_STATE_REGISTERING: Registering a new player. Waiting for the</span>
<span class="cm"> * e-mail address to be given</span>
<span class="cm"> * @WMUD_CLIENT_STATE_REGEMAIL_CONFIRM: E-mail address entered séms valid,</span>
<span class="cm"> * waiting for confirmation</span>
<span class="cm"> *</span>
<span class="cm"> * Game client states.</span>
<span class="cm"> */</span>
<span class="k">typedef</span><span class="w"> </span><span class="k">enum</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">WMUD_CLIENT_STATE_FRESH</span><span class="p">,</span>
<span class="w"> </span><span class="n">WMUD_CLIENT_STATE_PASSWAIT</span><span class="p">,</span>
<span class="w"> </span><span class="n">WMUD_CLIENT_STATE_MENU</span><span class="p">,</span>
<span class="w"> </span><span class="n">WMUD_CLIENT_STATE_INGAME</span><span class="p">,</span>
<span class="w"> </span><span class="n">WMUD_CLIENT_STATE_YESNO</span><span class="p">,</span>
<span class="w"> </span><span class="n">WMUD_CLIENT_STATE_REGISTERING</span><span class="p">,</span>
<span class="w"> </span><span class="n">WMUD_CLIENT_STATE_REGEMAIL_CONFIRM</span>
<span class="p">}</span><span class="w"> </span><span class="n">WmudClientState</span><span class="p">;</span>
<span class="n">GType</span><span class="w"> </span><span class="nf">wmud_client_state_get_type</span><span class="w"> </span><span class="p">(</span><span class="kt">void</span><span class="p">)</span><span class="w"> </span><span class="n">G_GNUC_CONST</span><span class="p">;</span>
<span class="cp">#define WMUD_TYPE_CLIENT_STATE (wmud_client_state_get_type())</span>
<span class="cp">#endif </span><span class="cm">/* __WMUD_CLIENT_STATE_H__ */</span>
</pre></div>
<div class="highlight"><pre><span></span><span class="cp">#include</span><span class="w"> </span><span class="cpf">"wmudclientstate.h"</span>
<span class="n">GType</span>
<span class="nf">wmud_client_state_get_type</span><span class="w"> </span><span class="p">(</span><span class="kt">void</span><span class="p">)</span>
<span class="p">{</span>
<span class="w"> </span><span class="k">static</span><span class="w"> </span><span class="k">volatile</span><span class="w"> </span><span class="n">gsize</span><span class="w"> </span><span class="n">g_define_type_id__volatile</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">g_once_init_enter</span><span class="p">(</span><span class="o">&</span><span class="n">g_define_type_id__volatile</span><span class="p">))</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">static</span><span class="w"> </span><span class="k">const</span><span class="w"> </span><span class="n">GEnumValue</span><span class="w"> </span><span class="n">values</span><span class="p">[]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">WMUD_CLIENT_STATE_FRESH</span><span class="p">,</span><span class="w"> </span><span class="s">"WMUD_CLIENT_STATE_FRESH"</span><span class="p">,</span><span class="w"> </span><span class="s">"fresh"</span><span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">WMUD_CLIENT_STATE_PASSWAIT</span><span class="p">,</span><span class="w"> </span><span class="s">"WMUD_CLIENT_STATE_PASSWAIT"</span><span class="p">,</span><span class="w"> </span><span class="s">"passwait"</span><span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">WMUD_CLIENT_STATE_MENU</span><span class="p">,</span><span class="w"> </span><span class="s">"WMUD_CLIENT_STATE_MENU"</span><span class="p">,</span><span class="w"> </span><span class="s">"menu"</span><span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">WMUD_CLIENT_STATE_INGAME</span><span class="p">,</span><span class="w"> </span><span class="s">"WMUD_CLIENT_STATE_INGAME"</span><span class="p">,</span><span class="w"> </span><span class="s">"ingame"</span><span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">WMUD_CLIENT_STATE_YESNO</span><span class="p">,</span><span class="w"> </span><span class="s">"WMUD_CLIENT_STATE_YESNO"</span><span class="p">,</span><span class="w"> </span><span class="s">"yesno"</span><span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">WMUD_CLIENT_STATE_REGISTERING</span><span class="p">,</span><span class="w"> </span><span class="s">"WMUD_CLIENT_STATE_REGISTERING"</span><span class="p">,</span><span class="w"> </span><span class="s">"registering"</span><span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">WMUD_CLIENT_STATE_REGEMAIL_CONFIRM</span><span class="p">,</span><span class="w"> </span><span class="s">"WMUD_CLIENT_STATE_REGEMAIL_CONFIRM"</span><span class="p">,</span><span class="w"> </span><span class="s">"regemail-confirm"</span><span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="nb">NULL</span><span class="p">,</span><span class="w"> </span><span class="nb">NULL</span><span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">};</span>
<span class="w"> </span><span class="n">GType</span><span class="w"> </span><span class="n">g_define_type_id</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">g_enum_register_static</span><span class="p">(</span><span class="n">g_intern_static_string</span><span class="p">(</span><span class="s">"WmudClientState"</span><span class="p">),</span><span class="w"> </span><span class="n">values</span><span class="p">);</span>
<span class="w"> </span><span class="n">g_once_init_leave</span><span class="p">(</span><span class="o">&</span><span class="n">g_define_type_id__volatile</span><span class="p">,</span><span class="w"> </span><span class="n">g_define_type_id</span><span class="p">);</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">g_define_type_id__volatile</span><span class="p">;</span>
<span class="p">}</span>
</pre></div>
<p>Still, it can be made more perfect by using the <a class="reference external" href="http://developer.gnome.org/gobject/stable/glib-mkenums.html">glib-mkenums</a> tool. I will read through the
GLib Makefiles tomorrow for some hints on this.</p>
<p>Edit: you can find the glib-mkenums solution <a class="reference external" href="blog/2014/8/16/registering-an-enum-type-in-glib-glib-mkenums-magic.html">here</a>.</p>
Development man pages on Fedora2013-01-05T18:20:41+00:002013-01-05T18:20:41+00:00Gergely Polonkaitag:None,2013-01-05:blog/2013/1/5/development-man-pages-on-fedora.html<p>If you use Fedora (like me), and can’t find the development manual pages for e.g. <tt class="docutils literal">printf(3)</tt>
(like me), just <tt class="docutils literal">yum install <span class="pre">man-pages</span></tt> (like me).</p>
Symfony 2 Configuration – Array of associative arrays2012-12-20T12:03:23+00:002012-12-20T12:03:23+00:00Gergely Polonkaitag:None,2012-12-20:blog/2012/12/20/symfony-2-configuration-array-of-associative-arrays.html<p>Few days ago I have struggled with a problem using Symfony2 configuration. I
wanted to add the following kind of configuration to <tt class="docutils literal">config.yml</tt>:</p>
<div class="highlight"><pre><span></span><span class="nt">acme_demo</span><span class="p">:</span>
<span class="w"> </span><span class="nt">transitions</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="p p-Indicator">{</span><span class="nt"> hc_cba</span><span class="p">:</span><span class="w"> </span><span class="nv">180</span><span class="w"> </span><span class="p p-Indicator">}</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="p p-Indicator">{</span><span class="nt"> cba_hc</span><span class="p">:</span><span class="w"> </span><span class="nv">-1</span><span class="w"> </span><span class="p p-Indicator">}</span>
</pre></div>
<p>The problem was that the stuff under <tt class="docutils literal">transitions</tt> is dynamic, so those <tt class="docutils literal">hc_cba</tt> and
<tt class="docutils literal">cba_hc</tt> tags can be pretty much …</p><p>Few days ago I have struggled with a problem using Symfony2 configuration. I
wanted to add the following kind of configuration to <tt class="docutils literal">config.yml</tt>:</p>
<div class="highlight"><pre><span></span><span class="nt">acme_demo</span><span class="p">:</span>
<span class="w"> </span><span class="nt">transitions</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="p p-Indicator">{</span><span class="nt"> hc_cba</span><span class="p">:</span><span class="w"> </span><span class="nv">180</span><span class="w"> </span><span class="p p-Indicator">}</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="p p-Indicator">{</span><span class="nt"> cba_hc</span><span class="p">:</span><span class="w"> </span><span class="nv">-1</span><span class="w"> </span><span class="p p-Indicator">}</span>
</pre></div>
<p>The problem was that the stuff under <tt class="docutils literal">transitions</tt> is dynamic, so those <tt class="docutils literal">hc_cba</tt> and
<tt class="docutils literal">cba_hc</tt> tags can be pretty much anything. After hitting many errors, I came to the solution:</p>
<div class="highlight"><pre><span></span><span class="cp"><?php</span>
<span class="nv">$rootNode</span>
<span class="o">-></span><span class="na">children</span><span class="p">()</span>
<span class="o">-></span><span class="na">arrayNode</span><span class="p">(</span><span class="s1">'state_machine'</span><span class="p">)</span>
<span class="o">-></span><span class="na">requiresAtLeastOneElement</span><span class="p">()</span>
<span class="o">-></span><span class="na">beforeNormalization</span><span class="p">()</span>
<span class="o">-></span><span class="na">ifArray</span><span class="p">()</span>
<span class="o">-></span><span class="na">then</span><span class="p">(</span><span class="k">function</span><span class="p">(</span><span class="nv">$values</span><span class="p">)</span> <span class="p">{</span>
<span class="nv">$ret</span> <span class="o">=</span> <span class="k">array</span><span class="p">();</span>
<span class="k">foreach</span> <span class="p">(</span><span class="nv">$values</span> <span class="k">as</span> <span class="nv">$value</span><span class="p">)</span> <span class="p">{</span>
<span class="k">foreach</span> <span class="p">(</span><span class="nv">$value</span> <span class="k">as</span> <span class="nv">$transition</span> <span class="o">=></span> <span class="nv">$time</span><span class="p">)</span> <span class="p">{</span>
<span class="nv">$ret</span><span class="p">[]</span> <span class="o">=</span> <span class="k">array</span><span class="p">(</span><span class="s1">'transition'</span> <span class="o">=></span> <span class="nv">$transition</span><span class="p">,</span> <span class="s1">'time'</span> <span class="o">=></span> <span class="nx">e</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">return</span> <span class="nv">$ret</span><span class="p">;</span>
<span class="p">})</span>
<span class="o">-></span><span class="na">end</span><span class="p">()</span>
<span class="o">-></span><span class="na">prototype</span><span class="p">(</span><span class="s1">'array'</span><span class="p">)</span>
<span class="o">-></span><span class="na">children</span><span class="p">()</span>
<span class="o">-></span><span class="na">scalarNode</span><span class="p">(</span><span class="s1">'transition'</span><span class="p">)</span><span class="o">-></span><span class="na">end</span><span class="p">()</span>
<span class="o">-></span><span class="na">scalarNode</span><span class="p">(</span><span class="s1">'time'</span><span class="p">)</span><span class="o">-></span><span class="na">end</span><span class="p">()</span>
<span class="o">-></span><span class="na">end</span><span class="p">()</span>
<span class="o">-></span><span class="na">end</span><span class="p">()</span>
<span class="o">-></span><span class="na">end</span><span class="p">()</span>
<span class="o">-></span><span class="na">end</span><span class="p">()</span>
<span class="p">;</span>
</pre></div>
Changing the session cookie’s name in Symfony 22012-10-13T12:49:28+00:002012-10-13T12:49:28+00:00Gergely Polonkaitag:None,2012-10-13:blog/2012/10/13/changing-the-session-cookie-s-name-in-symfony-2.html<p>I have a development server, on which I have several Symfony 2.x projects under the same hostname
in different directories. Now I’m facing a funny problem which is caused by that the cookies
Symfony places for each of my projects have the same name.</p>
<p>To change this, you …</p><p>I have a development server, on which I have several Symfony 2.x projects under the same hostname
in different directories. Now I’m facing a funny problem which is caused by that the cookies
Symfony places for each of my projects have the same name.</p>
<p>To change this, you will have to modify the <tt class="docutils literal">config.yml</tt> file like this:</p>
<div class="highlight"><pre><span></span><span class="nt">session</span><span class="p">:</span>
<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">SiteSpecificSessionName</span>
<span class="w"> </span><span class="nt">lifetime</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">3600</span>
</pre></div>
SmsGateway and SmsSender2012-10-07T00:10:26+00:002012-10-07T00:10:26+00:00Gergely Polonkaitag:None,2012-10-07:blog/2012/10/7/smsgateway-and-smssender.html<p>I have just uploaded my SmsGateway, SmsSender and SmsSenderBundle packages to <a class="reference external" href="http://github.com/gergelypolonkai">GitHub</a> and <a class="reference external" href="http://packagist.org">Packagist</a>. I hope some of you
will find it useful.</p>
<ul class="simple">
<li>SmsGateway<ul>
<li><a class="reference external" href="https://github.com/gergelypolonkai/smsgateway">SmsGateway on GitHub</a></li>
<li><a class="reference external" href="https://packagist.org/packages/gergelypolonkai/smsgateway">SmsGateway on Packagist</a></li>
</ul>
</li>
<li>SmsSender<ul>
<li><a class="reference external" href="https://github.com/gergelypolonkai/smssender">SmsSender on GitHub</a></li>
<li><a class="reference external" href="https://packagist.org/packages/gergelypolonkai/smssender">SmsSender on Packagist</a></li>
</ul>
</li>
<li>SmsSenderBundle<ul>
<li><a class="reference external" href="https://github.com/gergelypolonkai/smssender-bundle">SmsSenderBundle on GitHub</a></li>
<li><a class="reference external" href="https://packagist.org/packages/gergelypolonkai/smssender-bundle">SmsSenderBundle on Packagist</a></li>
</ul>
</li>
</ul>
Symfony 2 – Create role- and class-based ACLs with your roles coming from the ORM2012-09-16T18:39:25+00:002012-09-16T18:39:25+00:00Gergely Polonkaitag:None,2012-09-16:blog/2012/9/16/symfony-2-create-role-and-class-based-acls-with-your-roles-coming-from-the-orm.html<p>During the last weeks I had some serious issues with one of my private Symfony 2 projects. One of
my goals was to create a dynamic security system, e.g my administrators wanted to create roles,
and grant these roles access to different object types (classes) and/or objects.</p>
<p>So …</p><p>During the last weeks I had some serious issues with one of my private Symfony 2 projects. One of
my goals was to create a dynamic security system, e.g my administrators wanted to create roles,
and grant these roles access to different object types (classes) and/or objects.</p>
<p>So I have created a <tt class="docutils literal">User</tt> entity, which implements <tt class="docutils literal">UserInterface</tt> and
<tt class="docutils literal">AdvancedUserInterface</tt>, the latter for the possibility to enable/disable accounts and such. It
had a <tt class="docutils literal">$roles</tt> property, which was a <tt class="docutils literal">ManyToMany</tt> relation to the <tt class="docutils literal">Role</tt> entity, which
implemented <tt class="docutils literal">RoleInterface</tt>. Also I have created my own role hierarchy service that implements
<tt class="docutils literal">RoleHierarchyInterface</tt>.</p>
<p>So far so good, first tests. It soon turned out that if <tt class="docutils literal"><span class="pre">User::getRoles()</span></tt> returns a
<tt class="docutils literal">DoctrineCollection</tt> as it does by default, then the standard</p>
<div class="highlight"><pre><span></span><span class="cp"><?php</span>
<span class="nv">$this</span><span class="o">-></span><span class="na">get</span><span class="p">(</span><span class="s1">'security.context'</span><span class="p">)</span><span class="o">-></span><span class="na">isGranted</span><span class="p">(</span><span class="s1">'ROLE_ADMIN'</span><span class="p">);</span>
</pre></div>
<p>doesn’t work. I know, it should not be hard coded, as my roles and permission tables are dynamic,
I have just tested. So I fixed my <tt class="docutils literal">User</tt> entity so <tt class="docutils literal">getRoles()</tt> returns an array of <tt class="docutils literal">Role</tt>
objects instead of the <tt class="docutils literal">DoctrineCollection</tt>. Also I implemented a <tt class="docutils literal">getRolesCollection()</tt>
method to return the original collection, but I think it will never be used.</p>
<p>After that, I had to implement some more features so I put this task away. Then, I tried to
create my first <span class="caps">ACL</span>.</p>
<div class="highlight"><pre><span></span><span class="cp"><?php</span>
<span class="nv">$securityIdentity</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">RoleSecurityIdentity</span><span class="p">(</span><span class="s1">'ROLE_ADMIN'</span><span class="p">);</span>
<span class="nv">$objectIdentity</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">ObjectIdentity</span><span class="p">(</span><span class="s1">'newsClass'</span><span class="p">,</span> <span class="s1">'Acme\\DemoBundle\\Entity\\News'</span><span class="p">);</span>
<span class="nv">$acl</span> <span class="o">=</span> <span class="nv">$aclProvider</span><span class="o">-></span><span class="na">createAcl</span><span class="p">(</span><span class="nv">$objectIdentity</span><span class="p">);</span>
<span class="nv">$acl</span><span class="o">-></span><span class="na">insertClassAce</span><span class="p">(</span><span class="nv">$securityIdentity</span><span class="p">,</span> <span class="nx">MaskBuilder</span><span class="o">::</span><span class="na">MASK_OWNER</span><span class="p">);</span>
<span class="nv">$aclProvider</span><span class="o">-></span><span class="na">updateAcl</span><span class="p">(</span><span class="nv">$acl</span><span class="p">);</span>
</pre></div>
<p>I was about to check if the user who is logged in has an <tt class="docutils literal"><span class="caps">OWNER</span></tt> permission on the <tt class="docutils literal">User</tt>
class.</p>
<div class="highlight"><pre><span></span><span class="cp"><?php</span>
<span class="nv">$this</span><span class="o">-></span><span class="na">objectIdentity</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">ObjectIdentity</span><span class="p">(</span><span class="nx">self</span><span class="o">::</span><span class="na">OBJECT_ID</span><span class="p">,</span> <span class="nx">self</span><span class="o">::</span><span class="na">OBJECT_FQCN</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="nv">$this</span><span class="o">-></span><span class="na">securityContext</span><span class="o">-></span><span class="na">isGranted</span><span class="p">(</span><span class="s1">'OWNER'</span><span class="p">,</span> <span class="nv">$this</span><span class="o">-></span><span class="na">objectIdentity</span><span class="p">)</span> <span class="o">===</span> <span class="k">false</span><span class="p">)</span> <span class="p">{</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nx">AccessDeniedException</span><span class="p">(</span><span class="s1">'You don’t have the required permissions!'</span><span class="p">);</span>
<span class="p">}</span>
</pre></div>
<p>The <span class="caps">ACL</span> was defined based on a role, so everyone who had the <tt class="docutils literal">ROLE_ADMIN</tt> role should gain
access to the user listing page. But they didn’t. It took several weeks to find the cause, I
have put it on <a class="reference external" href="http://stackoverflow.com/questions/12057795/symfony-2-1-this-getsecurity-context-isgrantedrole-admin-returns-fa">stackoverflow</a>
and the Symfony Google Group, but no usable answers.</p>
<p>Then I went off for debugging. Setting up NetBeans for xdebug-based <span class="caps">PHP</span> debugging was real fun
under Fedora, but that’s another story. After a while I have found that Symfony’s basic access
decision manager checks for <tt class="docutils literal"><span class="pre">$role->getRole()</span></tt> only if <tt class="docutils literal">$role</tt> is an instance of
<tt class="docutils literal">Symfony\Component\Security\Core\Role\Role</tt>, instead of checking if the object implements
<tt class="docutils literal">Symfony\Component\Security\Core\Role\RoleInterface</tt>. So I’ve checked if the bug is already
reported. It turned out that it was, and my solution was available in a specific commit about a
year ago, but as <a class="reference external" href="https://github.com/symfony/symfony/commit/af70ac8d777873c49347ac828a817a400006cbea">Johannes Schmitt commented, it would introduce a security issue</a>, so it was
reverted. Unfortunately neither Johannes Schmitt, nor Fabien Potencier (nor anyone else) could (or
wanted) to tell about this issue. So the final (and somewhat hack-like) solution was to extend
<tt class="docutils literal">Symfony\Component\Security\Core\Role\Role</tt>. And boom! It worked.</p>
Do-Not-Track in IE10 vs. Apache2012-09-10T20:22:32+00:002012-09-10T20:22:32+00:00Gergely Polonkaitag:None,2012-09-10:blog/2012/9/10/do-not-track-in-ie10-vs-apache.html<p><a class="reference external" href="http://arstechnica.com/security/2012/09/apache-webserver-updated-to-ignore-do-not-track-settings-in-ie-10/">Apache developer decided not to accept Do-Not-Track headers from <span class="caps">IE10</span> users</a>,
because it’s enabled by default. So… if I install a plugin that hides the fact from the web
server that I’m using <span class="caps">IE10</span>, I become eligible of using it. But if I do this, I simply became …</p><p><a class="reference external" href="http://arstechnica.com/security/2012/09/apache-webserver-updated-to-ignore-do-not-track-settings-in-ie-10/">Apache developer decided not to accept Do-Not-Track headers from <span class="caps">IE10</span> users</a>,
because it’s enabled by default. So… if I install a plugin that hides the fact from the web
server that I’m using <span class="caps">IE10</span>, I become eligible of using it. But if I do this, I simply became
eligible because I consciously installed that addon, so I could actually use it without hiding the
fact. Sorry if I’m a bit Philosoraptorish…</p>
How to start becoming a web developer2012-09-07T18:12:12+00:002012-09-07T18:12:12+00:00Gergely Polonkaitag:None,2012-09-07:blog/2012/9/7/how-to-start-becoming-a-web-developer.html<p>A friend of mine asked me today how to become a web developer. It took me a while, but I made up
a checklist. It’s short, but it’s enough for the first steps.</p>
<div class="section" id="first-of-all-learn-english">
<h2>First of all, learn English</h2>
<p>Well, if you read this, maybe this was a bad …</p></div><p>A friend of mine asked me today how to become a web developer. It took me a while, but I made up
a checklist. It’s short, but it’s enough for the first steps.</p>
<div class="section" id="first-of-all-learn-english">
<h2>First of all, learn English</h2>
<p>Well, if you read this, maybe this was a bad first point…</p>
</div>
<div class="section" id="choose-a-language-and-stick-to-it">
<h2>Choose a language and stick to it!</h2>
<p>For the <span class="caps">UN</span>*X/Linux line, there is <span class="caps">PHP</span>. It’s free, easy to learn, and has many free tools and
documentations available. It can be used in a functional or an object-oriented way.</p>
<p>C# is another good way to start, but for the Windows line. It’s fully object oriented, and the
web is full of tutorials, how-tos and other resources.</p>
</div>
<div class="section" id="learn-the-basics-of-the-system-you-are-working-on">
<h2>Learn the basics of the system you are working on</h2>
<p>To become a good developer, learn at least the basics of the system you are working on. Basic
commands can always come in handy. Debugging (yes, you will do tons of bugs for sure) can become
much easier if you know the huge set of tools provided by your <span class="caps">OS</span>. You should also try to develop
in the chosen environment. Chose <span class="caps">PHP</span>? Get a Linux desktop! <span class="caps">ASP</span>.<span class="caps">NET</span>? Get a Windows. Everything
will be much easier!</p>
</div>
<div class="section" id="learn-the-basics-of-the-web-server-you-are-using">
<h2>Learn the basics of the web server you are using</h2>
<p><span class="caps">PHP</span> can run on <a class="reference external" href="http://httpd.apache.org/">Apache</a> (as a module), or any <span class="caps">CGI</span>-capable webserver,
like <a class="reference external" href="http://www.lighttpd.net/">lighttpd</a> or <a class="reference external" href="http://nginx.org/">nginx</a> (well, it can also run
on <span class="caps">IIS</span>, but trust me: you don’t want that). <span class="caps">ASP</span>.<span class="caps">NET</span> is designed for <span class="caps">IIS</span>, and although some
scripts can be run under a mono-capable server, it should still stay there.</p>
<p>Whichever you choose, learn the basics! How to start and stop the service,
basic configuration methods, modules/extensions, and so on. It’s more than sure
that you will face some issues while developing, so it can never hurt.</p>
</div>
<div class="section" id="keep-your-versions-under-control">
<h2>Keep your versions under control</h2>
<p>Version control is critical nowadays. It gives you a basic backup solution, can come in handy
with debugging, and if you ever want to work in a team, you will badly need it.</p>
<p>Subversion is a bit out of date now, and it’s kind of hard to set up.</p>
<p>Git is no easy. You will have to learn a lot of stuff, but basicly it’s just another version
control system. Just choose if you want to stick to merge-then-commit or rebase-then-commit, get
a client, and get on the run.</p>
<p>Microsoft’s Team Foundation is another good way if you are working in a team. It provides several
other features besides version controlling, and is well integrated into Visual Studio, which is
highly recommended for Windows based development.</p>
</div>
<div class="section" id="choose-an-environment-to-work-in">
<h2>Choose an environment to work in</h2>
<p>There are so many good tools out there. You should choose according to the language and <span class="caps">OS</span> on
what you are working on. <a class="reference external" href="http://www.zend.com/en/products/studio">Zend Studio</a> or <a class="reference external" href="https://netbeans.org/">Netbeans</a> are both good tools for <span class="caps">PHP</span> development, while <a class="reference external" href="http://www.visualstudio.com/">Visual Studio</a> is a best buy for Windows development. Both of these have many
ups and downs, but once you get in touch with their deeper parts, you will like them.</p>
</div>
Some thoughts about that dead Linux Desktop2012-09-05T09:01:31+00:002012-09-05T09:01:31+00:00Gergely Polonkaitag:None,2012-09-05:blog/2012/9/5/some-thoughts-about-that-dead-linux-desktop.html<p>There were some arguments in the near past on <a class="reference external" href="http://tirania.org/blog/archive/2012/Aug-29.html">What Killed the Linux Desktop</a>. After reading many replies, like <a class="reference external" href="http://www.zdnet.com/linus-torvalds-on-the-linux-desktops-popularity-problems-7000003641/">Linus
Torvalds’s</a>, I
have my own thoughts, too.</p>
<p>I know my place in the world, especially in the online community. I’m a Linux user for about 15
years and …</p><p>There were some arguments in the near past on <a class="reference external" href="http://tirania.org/blog/archive/2012/Aug-29.html">What Killed the Linux Desktop</a>. After reading many replies, like <a class="reference external" href="http://www.zdnet.com/linus-torvalds-on-the-linux-desktops-popularity-problems-7000003641/">Linus
Torvalds’s</a>, I
have my own thoughts, too.</p>
<p>I know my place in the world, especially in the online community. I’m a Linux user for about 15
years and a Linux administrator for 10 years now, beginning with WindowMaker and something that I
remember as <span class="caps">GNOME</span> without a version number. I have committed some minor code chunks and
translations in some minor projects, so I’m not really into it from the “write” side (well, until
now, since I have began to write this blog, and much more, but don’t give a penny for my words
until you see it).</p>
<p>I’m using Linux since 2.2 and <span class="caps">GNOME</span> since 1.whatever. It’s nice that a program compiled years ago
still runs on today’s Linux kernel, especially if you see old <span class="caps">DOS</span>/Windows software failing to
start on a new Windows 7 machine. I understand Linus’ point that breaking external APIs is bad,
and I think it can work well on the kernel’s level. But the desktop level is much different. As
the Linux Desktop has such competitors (like <span class="caps">OS</span>/X and Windows’ Aero and Metro), they have to give
something new to the users almost every year to keep up with them. Eye candies are a must (yes,
of course my techy fellows, they are worthless, but users <em>need</em> it), and they can not be created
without extending APIs. And the old <span class="caps">API</span>… well, it fades away fast. I don’t really understand
however, why they have to totally disappear, like <a class="reference external" href="http://developer.gnome.org/gtk/stable/GtkDialog.html#GtkDialogFlags">GTK_DIALOG_NO_SEPARATOR</a> in Gtk+3. It could be
replaced with a 0 value (e.g: it won’t do anything). This way my old Gtk+2 program could compile
with Gtk+3 nicely. Also, there could be a small software that goes through your source code and
warn you about such deprecated (and no-doer but still working) things. Porting applications
between Gtk+ (and thus, <span class="caps">GNOME</span>) versions became a real pain, which makes less enthusiast
programmers stop developing for Linux. Since I’m a <span class="caps">GNOME</span> guy for years, I can tell nothing about
Qt and <span class="caps">KDE</span>, but for the <span class="caps">GNOME</span> guys, this is a bad thing. As of alternatives, there is Java. No,
wait… it turned out recently that <a class="reference external" href="http://www.theregister.co.uk/2012/08/31/critical_flaw_found_in_patched_java">it has several security bugs</a>. Also it’s not
that multiplatform as they say (I can’t find the article on that at the moment, but I have proof).
Also, the JVMs out there eat up so much resources, which makes it a bit hard and expensive to use.</p>
<p>Also, I see another problem: those blasted package managers. <span class="caps">RPM</span>, <span class="caps">DPKG</span>, Portage, whatever. What
the hell? Why are there so many? Why do developers reinvent the wheel? The nave is too small or
there are to few spokes? Come on… we live in an open source world! Contribute to the one and
only package manager (which one is that I don’t actually care)! I’m sure the two (three, many)
bunches of develoeprs could make a deal. Thus, it could become better and “outsider” companies
would be happier to distribute their software for Linux platforms.</p>
<p>And now that we get to the big companies. I don’t really understand them. nVidia and <span class="caps">ATI</span> made
their own closed source drivers for Linux. Some other hardware vendors also write Linux drivers,
and as the kernel <span class="caps">API</span> doesn’t really change, they will work for a long time. But what about
desktop application vendors? Well, they try to stick to a desktop environment or two, and if they
change too frequently, they stop developing for Linux, like Skype did (<span class="caps">OK</span>, maybe Skype has other
reasons, but you see my point). But why? The main part for Linux programs is the Linux kernel
and the basic userland like libc/stdlib++. If you write graphical software, it will have to use
X-Windows. Yes, it’s much different in many ways, mostly because they have a… well… pretty ugly
design by default. But still, it’s the same on every Linux distributions, as it became somewhat
an industry standard, as it was already on the market back in the old <span class="caps">UN</span>*X days. The protocol
itself changed just like the Linux kernel: almost no change at all, just some new features.</p>
<p>So what kills the Linux desktop in my opinion is these constant wars inside, and the lack of
support from the outside. Open Source is good, but until these (mostly the first) problems are
not resolved, Linux Desktop can do nothing on the market. It’s a downward spiral hard to escape.</p>
Upgrades requiring a reboot on Linux? At last!2012-06-22T20:04:51+00:002012-06-22T20:04:51+00:00Gergely Polonkaitag:None,2012-06-22:blog/2012/6/22/upgrades-requiring-a-reboot-on-linux-at-last.html<p>I’ve recently received an article on Google+ about Fedora’s new idea: package upgrades that
require a reboot. The article said that Linux guys have lost their primary adoo: “Haha! I don’t
have to reboot my system to install system upgrades!” My answer was always this: “Well, actually …</p><p>I’ve recently received an article on Google+ about Fedora’s new idea: package upgrades that
require a reboot. The article said that Linux guys have lost their primary adoo: “Haha! I don’t
have to reboot my system to install system upgrades!” My answer was always this: “Well, actually
you should…”</p>
<p>I think this can be a great idea if distros implement it well. PackageKit was a good first step
on this road. That software could easily solve such an issue. However, it is sooo easy to do it
wrong. The kernel, of course, can not be upgraded online (or could it be? I have some theories on
this subject, wonder if it can be implemented…), but other packages are much different. From the
users’ point of view the best would be if the packages would be upgraded in the background
seemlessly. E.g. PackageKit should check if the given executable is running. If not, it should
upgrade it, while notifying the user like “Hey dude, don’t start Anjuta now, I’m upgrading it!”,
or simply denying to start it. Libraries are a bit different, as PackageKit should check if any
running executables are using the library. Meanwhile, <span class="caps">PK</span> should also keep a notification
somewhere telling the users that some packages could be upgraded, but without stopping
this-and-that, it can not be done.</p>
<p>I know these things are easier said than done. But I think (a) users should tell such ideas to
the developers and (b) developers (mostly large companies, like Microsoft or Apple) should listen
to them, and at least think of these ideas. Some users are not as stupid as they think…</p>
SSH login FAILed on Red Had Enterprise Linux 6.22012-06-18T18:28:45+00:002012-06-18T18:28:45+00:00Gergely Polonkaitag:None,2012-06-18:blog/2012/6/18/ssh-login-failed-on-red-hat-enterprise-linux-6-2.html<p>Now this was a mistake I should not have done…</p>
<p>About a month ago I have moved my <span class="caps">AWS</span> <span class="caps">EC2</span> machine from Amazon Linux to <span class="caps">RHEL</span> 6.2. This was good.
I have moved all my files and stuff, recreated my own user, everything was just fine. Then I
copied …</p><p>Now this was a mistake I should not have done…</p>
<p>About a month ago I have moved my <span class="caps">AWS</span> <span class="caps">EC2</span> machine from Amazon Linux to <span class="caps">RHEL</span> 6.2. This was good.
I have moved all my files and stuff, recreated my own user, everything was just fine. Then I
copied my <a class="reference external" href="https://github.com/tv42/gitosis">gitosis</a> account (user <tt class="docutils literal">git</tt> and its home
directory). Then I tried to log in. It failed. I was blaming OpenSSH for a week or so, changed
the config file in several ways, tried to change the permissions on <tt class="docutils literal"><span class="pre">~git/.ssh/*</span></tt>, but still
nothing. Permission were denied, I was unable to push any of my development changes. Now after a
long time of trying, I coincidently <tt class="docutils literal">tail <span class="pre">-f</span></tt>-ed <tt class="docutils literal">/var/log/audit/audit.log</tt> (wanted to open
<tt class="docutils literal">auth.log</tt> instead) and that was my first good point. It told me that <tt class="docutils literal">sshd</tt> was unable to
read <tt class="docutils literal"><span class="pre">~git/.ssh/authorized_keys</span></tt>, which gave me the idea to run <tt class="docutils literal">restorecon</tt> on <tt class="docutils literal">/home/git</tt>.
It solved the problem.</p>
<p>All hail SELinux and <span class="caps">RBAC</span>!</p>
Wordpress madness2012-06-14T06:40:12+00:002012-06-14T06:40:12+00:00Gergely Polonkaitag:None,2012-06-14:blog/2012/6/14/wordpress-madness.html<p>I’m a bit fed up that I had to install <a class="reference external" href="http://www.mysql.com/">MySQL</a> on my server to have
<a class="reference external" href="http://wordpress.org/">Wordpress</a> working, so I’ve Googled a bit to find a solution for my
pain. I found <a class="reference external" href="http://codex.wordpress.org/Using_Alternative_Databases">this</a>. I don’t know
when this post was written, but I think it’s a …</p><p>I’m a bit fed up that I had to install <a class="reference external" href="http://www.mysql.com/">MySQL</a> on my server to have
<a class="reference external" href="http://wordpress.org/">Wordpress</a> working, so I’ve Googled a bit to find a solution for my
pain. I found <a class="reference external" href="http://codex.wordpress.org/Using_Alternative_Databases">this</a>. I don’t know
when this post was written, but I think it’s a bit out of date. I mean come on, <span class="caps">PDO</span> is the part
of <span class="caps">PHP</span> for ages now, and they say adding a <span class="caps">DBAL</span> to the dependencies would be a project as large as
(or larger than) <span class="caps">WP</span> itself. Well, yes, but <span class="caps">PHP</span> is already a dependency, isn’t it? Remove it
guys, it’s too large!</p>
<p>Okay, to be serious… Having a heavily MySQL dependent codebase is a bad thing in my opinion, and
changing it is no easy task. But once it is done, it would be a child’s play to keep it up to
date, and to port <span class="caps">WP</span> to other database backends. And it would be more than enough to call it 4.0,
and raising version numbers fast is a must nowadays (right, Firefox and Linux Kernel guys?)</p>
Fast world, fast updates2012-03-27T06:18:43+00:002012-03-27T06:18:43+00:00Gergely Polonkaitag:None,2012-03-27:blog/2012/3/27/fast-world-fast-updates.html<p>We live in a fast world, that’s for sure. When I first heard about Ubuntu Linux and their goals,
I was happy: they gave a Debian to everyone, but in different clothes. It had fresh software in
it, and even they gave support of a kind. It was easy …</p><p>We live in a fast world, that’s for sure. When I first heard about Ubuntu Linux and their goals,
I was happy: they gave a Debian to everyone, but in different clothes. It had fresh software in
it, and even they gave support of a kind. It was easy to install and use, even if one had no
Linux experience before. So people liked it. I’ve even installed it on some of my servers
because of the new package versions that came more often. Thus I got an up to date system.
However, it had a price. After a while, security updates came more and more often, and when I had
a new critical update every two or three days, I’ve decided to move back to Debian. Fortunately I
did this at the time of a new release, so I didn’t really loose any features.</p>
<p>After a few years passed, even Debian is heading this very same way. But as I see, the cause is
not the same. It seems that upstream software is hitting these bugs, and even the Debian guys
don’t have the time to check for them. At the time of a <span class="caps">GNOME</span> version bump (yes, <span class="caps">GNOME</span> 3 is a
really big one for the <span class="caps">UN</span>*X-like OSes), when hundreds of packages need to be checked, security
bugs show off more often. On the other hand however, Debian is releasing a new security update
every day (I had one on each of the last three days). This, of course, is good from one point of
view as we get a system that is more secure, but most administrators don’t have maintenance
windows this often. I can think of some alternatives like Fedora, but do I really have to change?
Dear fellow developers, please code more carefully instead!</p>
PHP 5.4 released2012-03-20T13:31:12+00:002012-03-20T13:31:12+00:00Gergely Polonkaitag:None,2012-03-20:blog/2012/3/20/php-5-4-released.html<p>After a long time of waiting, <span class="caps">PHP</span> announced 5.4 release on 1 March (also, today they announced
that they finally migrate to Git, which is sweet from my point of view, but it doesn’t really matter).</p>
<p>About a year ago we became very agressive towards a developer who …</p><p>After a long time of waiting, <span class="caps">PHP</span> announced 5.4 release on 1 March (also, today they announced
that they finally migrate to Git, which is sweet from my point of view, but it doesn’t really matter).</p>
<p>About a year ago we became very agressive towards a developer who created our internal e-learning
system. Their database was very insecure, and they didn’t really follow industry standards in
many ways. Thus, we forced them to move from Windows + Apache 2.0 + <span class="caps">PHP</span> 5.2 + MySQL 4.0 to Debian
Linux 6.0 + Apache 2.2 + <span class="caps">PHP</span> 5.3 + MySQL 5.1. It was fun (well, from our point of view), as their
coders… well… they are not so good. The code that ran “smoothly” on the old system failed at many
points on the new one. So they code and code, and write more code. And they still didn’t finish.
And now 5.4 is here. Okay, I know it will take some time to get into the Debian repositories, but
it’s here. And they removed <tt class="docutils literal">register_globals</tt>, which will kill that funny code again at so
many points that they will soon get to rewrite the whole code to make it work. And I just sit
here in my so-much-comfortable chair, and laugh. Am I evil?</p>
Why you should always test your software with production data2011-12-11T12:14:51+00:002011-12-11T12:14:51+00:00Gergely Polonkaitag:None,2011-12-11:blog/2011/12/11/why-you-should-always-test-your-software-with-production-data.html<p>I’m writing a software for my company in <span class="caps">PHP</span>, using the Symfony 2 framework. I’ve finished all
the work, created some sample data, it loaded perfectly. Now I put the whole thing into
production and tried to upload the production data into it. Guess what… it didn’t …</p><p>I’m writing a software for my company in <span class="caps">PHP</span>, using the Symfony 2 framework. I’ve finished all
the work, created some sample data, it loaded perfectly. Now I put the whole thing into
production and tried to upload the production data into it. Guess what… it didn’t load.</p>
Inverse of sort2011-09-18T14:57:31+00:002011-09-18T14:57:31+00:00Gergely Polonkaitag:None,2011-09-18:blog/2011/9/18/inverse-of-sort.html<p>I’m using *<span class="caps">NIX</span> systems for about 14 years now, but it can still show me new things. Today I had
to generate a bunch of random names. I’ve create a small perl script which generates permutations
of some usual Hungarian first and last names, occasionally prefixing it with …</p><p>I’m using *<span class="caps">NIX</span> systems for about 14 years now, but it can still show me new things. Today I had
to generate a bunch of random names. I’ve create a small perl script which generates permutations
of some usual Hungarian first and last names, occasionally prefixing it with a ‘Dr.’ title or
using double first names. For some reasons I forgot to include uniqueness check in the script.
When I ran it in the command line, I realized the mistake, so I appended <tt class="docutils literal">| sort | uniq</tt> to the
command line. So I had around 200 unique names, but in alphabetical order, which was awful for my
final goal. Thus, I tried shell commands like <tt class="docutils literal">rand</tt> to create a random order, and when many of
my tries failed, the idea popped in my mind (not being a native English speaker): “I don’t have to
create »random order«, but »shuffle the list«. So I started typing <tt class="docutils literal">shu</tt>, pressed Tab in the
Bash shell, and voilà! <tt class="docutils literal">shuf</tt> is the winner, it does just exactly what I need:</p>
<blockquote>
<dl class="docutils">
<dt><strong><span class="caps">NAME</span></strong></dt>
<dd>shuf - generate random permutations</dd>
</dl>
</blockquote>
<p>Thank you, Linux Core Utils! :)</p>
Proxy only non-existing files with mod_proxy and mod_rewrite2011-06-10T14:20:43+00:002011-06-10T14:20:43+00:00Gergely Polonkaitag:None,2011-06-10:blog/2011/6/10/proxy-only-non-existing-files-with-mod-proxy-and-mod-rewrite.html<p>Today I got an interesting task. I had to upload some pdf documents to a site. The domain is
ours, but we don’t have access to the application server that is hosting the page yet. Until we
get it in our hands, I did a trick.</p>
<p>I enabled <cite>mod_rewrite …</cite></p><p>Today I got an interesting task. I had to upload some pdf documents to a site. The domain is
ours, but we don’t have access to the application server that is hosting the page yet. Until we
get it in our hands, I did a trick.</p>
<p>I enabled <cite>mod_rewrite</cite>, <cite>mod_proxy</cite> and <cite>mod_proxy_http</cite>, then added the following lines to my
apache config:</p>
<div class="highlight"><pre><span></span><span class="nb">RewriteEngine</span><span class="w"> </span><span class="k">on</span>
<span class="nb">RewriteRule</span><span class="w"> </span>^/$<span class="w"> </span>http://172.16.72.131:8080/<span class="w"> </span>[QSA,L,P]
<span class="nb">RewriteCond</span><span class="w"> </span>%{REQUEST_FILENAME}<span class="w"> </span>!-f
<span class="nb">RewriteRule</span><span class="w"> </span>^/(.*)<span class="w"> </span>http://172.16.72.131:8080/$1<span class="w"> </span>[QSA,L,P]
<span class="nb">Order</span><span class="w"> </span>allow,deny
<span class="nb">Allow</span><span class="w"> </span>from<span class="w"> </span><span class="k">all</span>
</pre></div>
<p>I’m not totally sure it’s actually secure, but it works for now.</p>
Oracle Database “incompatible” with Oracle Linux?2011-05-27T17:53:31+00:002011-05-27T17:53:31+00:00Gergely Polonkaitag:None,2011-05-27:blog/2011/5/27/oracle-database-incompatible-with-oracle-linux.html<p>Today I gave a shot to install <a class="reference external" href="http://www.oracle.com/us/technologies/linux/overview/index.html">Oracle Linux</a>. I thought I could easily
install an Oracle <span class="caps">DBA</span> on it. Well, I was naive.</p>
<p>As only the 5.2 version is supported by XenServer 5.5, I downloaded that version of Oracle Linux.
Installing it was surprisingly fast and easy …</p><p>Today I gave a shot to install <a class="reference external" href="http://www.oracle.com/us/technologies/linux/overview/index.html">Oracle Linux</a>. I thought I could easily
install an Oracle <span class="caps">DBA</span> on it. Well, I was naive.</p>
<p>As only the 5.2 version is supported by XenServer 5.5, I downloaded that version of Oracle Linux.
Installing it was surprisingly fast and easy, it asked almost nothing, and booted without any problems.</p>
<p>After this came the <span class="caps">DBA</span>, 10.2, which bloated an error message in my face saying that this is an
unsupported version of Linux. Bah.</p>
<p>Is it only me, or is it really strange that Oracle doesn’t support their own distro?</p>
Citrix XenServer 5.5 vs. Debian 5.0 upgrade to 6.02011-05-27T17:33:41+00:002011-05-27T17:33:41+00:00Gergely Polonkaitag:None,2011-05-27:blog/2011/5/27/citrix-xenserver-vs-debian-5-0-upgrade-to-6-0.html<p>Few weeks ago I’ve upgraded two of our Debian based application servers from 5.0 to 6.0.
Everything went fine, as the upgraded packages worked well with the 4.2 JBoss instances. For the
new kernel we needed a reboot, but as the network had to be rebuilt …</p><p>Few weeks ago I’ve upgraded two of our Debian based application servers from 5.0 to 6.0.
Everything went fine, as the upgraded packages worked well with the 4.2 JBoss instances. For the
new kernel we needed a reboot, but as the network had to be rebuilt, I postponed this reboot until
the network changes. With the network, everything went fine again, we successfully migrated our
mail servers behind a firewall. Also the Xen server (5.5.0, upgrade to 5.6 still has to wait for
a week or so) revolted well with some storage disks added. But the application servers remained silent…</p>
<p>After checking the console, I realised that they don’t have an active console. And when I tried
to manually start them, XenServer refused with a message regarding <tt class="docutils literal">pygrub</tt>.</p>
<p>To understand the problem, I had to understand how XenServer boots Debian. It reads the
<tt class="docutils literal">grub.conf</tt> on the first partition’s root or <tt class="docutils literal">/boot</tt> directory, and starts the first option,
without asking (correct me, if I’m mistaken somewhere). However, this <tt class="docutils literal">pygrub</tt> thing can not
parse the new, grub2 config. This is kinda frustrating.</p>
<p>For the first step, I quickly installed a new Debian 5.0 system from my template. Then I attached
the disks of the faulty virtual machine, and mounted all its partitions. This way I could reach
my faulty 6.0 system with a chroot shell, from which I could install the <tt class="docutils literal"><span class="pre">grub-legacy</span></tt> package
instead of grub, install the necessary kernel and XenServer tools (which were missing from both
machines somehow), then halt the rescue system, and start up the original instance.</p>
<p>Next week I will do an upgrade on the XenServer to 5.6.1. I hope no such problems will occur.</p>
Gentoo hardened desktop with GNOME 3 – Round two2011-05-18T10:28:14+00:002011-05-18T10:28:14+00:00Gergely Polonkaitag:None,2011-05-18:blog/2011/5/18/gentoo-hardened-desktop-with-gnome-3-round-two.html<p>After several hours of <tt class="docutils literal">package.keywords</tt>/<tt class="docutils literal">package.use</tt> editing and package compiling, I
managed to install <span class="caps">GNOME</span> 3 on my notebook. Well, I mean, the <span class="caps">GNOME</span> 3 packages. Unfortunately the
<tt class="docutils literal">fglrx</tt> driver didn’t seem to recognise my <span class="caps">ATI</span> Mobility <span class="caps">M56P</span> card, and the open source driver
didn’t want …</p><p>After several hours of <tt class="docutils literal">package.keywords</tt>/<tt class="docutils literal">package.use</tt> editing and package compiling, I
managed to install <span class="caps">GNOME</span> 3 on my notebook. Well, I mean, the <span class="caps">GNOME</span> 3 packages. Unfortunately the
<tt class="docutils literal">fglrx</tt> driver didn’t seem to recognise my <span class="caps">ATI</span> Mobility <span class="caps">M56P</span> card, and the open source driver
didn’t want to give me <span class="caps">GLX</span> support. When I finally found some clues on what should I do, I had to
use my notebook for work, so I installed Fedora 14 on it. Then I realised that <span class="caps">GNOME</span> 3 is already
included in Rawhide (Fedora 15), so I quickly downloaded and installed that instead. Now I have
to keep this machine in a working state for a few days, so I will learn SELinux stuff in its
native environment.</p>
<p>When I installed Fedora 14, the first <span class="caps">AVC</span> message popped up after about ten minutes. That was a
good thing, as I wanted to see <tt class="docutils literal">setroubleshoot</tt> in action. However, in Fedora 15, the <span class="caps">AVC</span>
bubbles didn’t show up even after a day. I raised my left eyebrow and said that’s impossible,
SELinux must be disabled. And it’s not! It’s even in enforcing mode! And it works just fine. I
like it, and I hope I will be able to get the same results with Gentoo if I can get back to testing…</p>
Zabbix performance tip2011-05-13T19:03:31+00:002011-05-13T19:03:31+00:00Gergely Polonkaitag:None,2011-05-13:blog/2011/5/13/zabbix-performance-tip.html<p>Recently I have switched from <a class="reference external" href="http://oss.oetiker.ch/mrtg/"><span class="caps">MRTG</span></a> + <a class="reference external" href="http://www.cacti.net/">Cacti</a> + <a class="reference external" href="http://www.nagios.org/">Nagios</a> + <a class="reference external" href="http://www.gnokii.org/">Gnokii</a> to <a class="reference external" href="http://www.zabbix.com/">Zabbix</a>, and I must say I’m more than
satisfied with it. It can do anything the former tools did, and much more. First of all, it can
do the same monitoring as Nagios did, but it does much more …</p><p>Recently I have switched from <a class="reference external" href="http://oss.oetiker.ch/mrtg/"><span class="caps">MRTG</span></a> + <a class="reference external" href="http://www.cacti.net/">Cacti</a> + <a class="reference external" href="http://www.nagios.org/">Nagios</a> + <a class="reference external" href="http://www.gnokii.org/">Gnokii</a> to <a class="reference external" href="http://www.zabbix.com/">Zabbix</a>, and I must say I’m more than
satisfied with it. It can do anything the former tools did, and much more. First of all, it can
do the same monitoring as Nagios did, but it does much more fine. It can check several parameters
within one request, so network traffic is kept down. Also, its web front-end can generate any
kinds of graphs from the collected data, which took Cacti away. Also, it can do <span class="caps">SNMP</span> queries
(v1-v3), so querying my switches’ port states and traffic made easy, taking <span class="caps">MRTG</span> out of the
picture (I know Cacti can do it either, it had historical reasons we had both tools installed).
And the best part: it can send <span class="caps">SMS</span> messages via a <span class="caps">GSM</span> modem natively, while Nagios had to use
Gnokii. The trade-off is, I had to install Zabbix agent on all my monitored machines, but I think
it worths the price. I even have had to install <span class="caps">NRPE</span> to monitor some parameters, which can be a
pain on Windows hosts, while Zabbix natively supports Windows, Linux and Mac <span class="caps">OS</span>/X.</p>
<p>So I only had to create a MySQL database (which I already had for <span class="caps">NOD32</span> central management), and
install Zabbix server. Everything went fine, until I reached about 1300 monitored parameters.
MySQL seemed to be a bit slow on disk writes, so my Zabbix “queue” filled up in no time. After
reading some forums, I decided to switch to PostgreSQL instead. Now it works like charm, even
with the default Debian settings. However, I will have to add several more parameters, and my
boss wants as many graphs as you can imagine, so I’m more than sure that I will have to fine tune
my database later.</p>
Ethical Hacking 20122011-05-12T20:54:42+00:002011-05-12T20:54:42+00:00Gergely Polonkaitag:None,2011-05-12:blog/2011/5/12/ethical-hacking-2011.html<p>Today I went to the Ethical Hacking conference with my boss. It was my first appearance at such
conferences, but I hope there will be more. Although we just started to redesign our <span class="caps">IT</span> security
infrastructure with a 90% clear goal, it was nice to hear that everything is vulnerable …</p><p>Today I went to the Ethical Hacking conference with my boss. It was my first appearance at such
conferences, but I hope there will be more. Although we just started to redesign our <span class="caps">IT</span> security
infrastructure with a 90% clear goal, it was nice to hear that everything is vulnerable. I was
thinking if we should sell all our <span class="caps">IT</span> equipments, fire all our colleagues (you know, to prevent
social engineering), and move to the South Americas to herd llamas or sheep, so the only danger
would be some lurking pumas or jaguars. Or I simply leave my old background image on my desktop,
from the well-known game, which says: Trust is a weakness.</p>
<p>Anyways, the conference was really nice. We heard about the weaknesses of Android, Oracle, and
even FireWire. They showed some demos about everything, exploited some free and commercial
software with no problem at all. We have seen how much power the virtualisation admin has
(although I think it can be prevented, but I’m not sure yet). However, in the end, we could see
that the Cloud is secure (or at least it can be, in a few months or so), so I’m not totally
pessimistic. See you next time at Hacktivity!</p>
Gentoo hardened desktop with GNOME 3 – Round one2011-05-12T20:32:41+00:002011-05-12T20:32:41+00:00Gergely Polonkaitag:None,2011-05-12:blog/2011/5/12/gentoo-hardened-desktop-with-gnome-3-round-one.html<p>After having some hard times with Ubuntu (upgrading from 10.10 to 11.04), I decided to switch back
to my old friend, Gentoo. As I’m currently learning about Linux hardening, I decided to use the
new SELinux profile, which supports the v2 reference policy.</p>
<p>Installation was pretty easy …</p><p>After having some hard times with Ubuntu (upgrading from 10.10 to 11.04), I decided to switch back
to my old friend, Gentoo. As I’m currently learning about Linux hardening, I decided to use the
new SELinux profile, which supports the v2 reference policy.</p>
<p>Installation was pretty easy, using the <a class="reference external" href="http://www.gentoo.org/doc/hu/handbook/handbook-x86.xml">Gentoo x86 Handbook</a>. This profile automatically turns on
the <tt class="docutils literal"><span class="caps">USE</span>=selinux</tt> flag (so does the old SELinux profile), but deprecated <tt class="docutils literal"><span class="caps">FEATURE</span>=loadpolicy</tt>
(which is turned on by the profile, so portage will complain about it until you disable it in
<tt class="docutils literal">/etc/make.conf</tt>).</p>
<p>For the kernel, I chose <tt class="docutils literal"><span class="pre">hardened-sources-2.6.37-r7</span></tt>. This seems to be recent enough for my
security testing needs. I turned on both SELinux, PaX and grsecurity. So far, I have no problem
with it, but I don’t have X installed yet, which will screw up things for sure.</p>
<p>After having those hard times with Ubuntu mentioned before, I decided not to install Grub2 yet, as
it renders things unusable (eg. my Windows 7 installation, which I sometimes need at the office).
So I installed Grub 0.97 (this is the only version marked as stable, as I remember), touched
<tt class="docutils literal">/.autorelabel</tt>, and reboot.</p>
<p>My first mistake was using an <span class="caps">UUID</span> as the root device on the kernel parameter list (I don’t want
to list all the small mistakes like forgetting to include to correct <span class="caps">SATA</span> driver from my kernel
and such). Maybe I was lame, but after including <tt class="docutils literal">/dev/sda5</tt> instead of the <span class="caps">UUID</span> thing, it
worked like…</p>
<p>Well, charm would not be the good word. For example, I forgot to install the <tt class="docutils literal">lvm2</tt> package, so
nothing was mounted except my root partition. After I installed it with the install <span class="caps">CD</span>, I assumed
everything will be all right, but I was wrong.</p>
<p><tt class="docutils literal">udev</tt> and <span class="caps">LVM</span> is a critical point in a hardened environment. <tt class="docutils literal">udev</tt> itself doesn’t want to
work without the <tt class="docutils literal">CONFIG_DEVFS_TEMPFS=y</tt> kernel option, so I also had to change that. It seemed
that it can be done without the install <span class="caps">CD</span>, as it compiled the kernel with no problems. However,
when it reached the point when it compresses the kernel with gzip, it stopped with a <tt class="docutils literal">Permission
denied</tt> message (although it was running with root privileges).</p>
<p>The most beautiful thing in the hardened environment with Mandatory Access Control enabled is that
root is not a real power user any more by default. You can get this kind of messages many times.
There are many tools to debug these, I will talk about these later.</p>
<p>So, my <tt class="docutils literal">gzip</tt> needed a fix. After digging a bit on the Internet, I found that the guilty thing
is text relocation, which can be corrected if <tt class="docutils literal">gzip</tt> is compiled with <span class="caps">PIC</span> enabled. Thus, I
turned on <tt class="docutils literal"><span class="caps">USE</span>=pic</tt> flag globally, and tried to remerge gzip. Of course it failed, as it had to
use gzip to unpack the gzip sources. So it did when I tried to install the PaX tools and
<tt class="docutils literal">gradm</tt> to turn these checks off. The install <span class="caps">CD</span> came to the rescue again, with which I
successfully recompiled gzip, and with this new gzip, I compressed my new kernel, with which udev
started successfully. So far, so good, let’s try to reboot!</p>
<p>Damn, <span class="caps">LVM</span> is still not working. So I decided to finally consult the Gentoo hardened guide. It
says that the <span class="caps">LVM</span> startup scripts under <tt class="docutils literal">/lib/rcscripts/…</tt> must be modified, so <span class="caps">LVM</span> will put its
lock files under <tt class="docutils literal">/etc/lvm/lock</tt> instead of <tt class="docutils literal"><span class="pre">/dev/.lvm</span></tt>. After this step and a reboot, <span class="caps">LVM</span>
worked fine (finally).</p>
<p>The next thing was the file system labelling. SELinux should automatically relabel the entire
file system at boot time whenever it finds the <tt class="docutils literal">/.autorelabel</tt> file. Well, in my case it didn’t
happen. After checking the <a class="reference external" href="http://wiki.gentoo.org/wiki/Hardened_Gentoo">Gentoo Hardening</a>
docs, I realised that the <tt class="docutils literal">rlpkg</tt> program does exactly the same (as far as I know, it is
designed specifically for Gentoo). So I ran <tt class="docutils literal">rlpkg</tt>, and was kind of shocked. It says it will
relabel ext2, ext3, xfs and <span class="caps">JFS</span> partitions. Oh great, no ext4 support? Well, consulting the
forums and adding some extra lines to <tt class="docutils literal">/etc/portage/package.keywords</tt> solved the problem
(<tt class="docutils literal">rlpkg</tt> and some dependencies had to have the <tt class="docutils literal">~x86</tt> keyword set). Thus, <tt class="docutils literal">rlpkg</tt>
relabelled my file systems (I checked some directories with <tt class="docutils literal">ls <span class="pre">-lZ</span></tt>, it seemed good for me).</p>
<p>Now it seems that everything is working fine, except the tons of audit messages. Tomorrow I will
check them with <tt class="docutils literal">audit2why</tt> or <tt class="docutils literal">audit2allow</tt> to see if it is related with my SELinux lameness,
or with a bug in the policy included with Gentoo.</p>