Blogs | blog.moe.ph

XLSXWriter Pandas CSV Python Notes

Sun, 10 Jan 2021 00:00:00 +0000

Some useful notes in dealing with xlsxwriter, csv, and pandas thing. Still planning to add sql support for fetching data from remote source, then processing it via pandas, then save as xlsx data.

import csv
import xlsxwriter
import datetime as dt
import pandas as pd

# Create an new Excel file and add a worksheet.
workbook = xlsxwriter.Workbook('sample-xlsx.xlsx')
worksheet = workbook.add_worksheet('GPUH')
date_format = workbook.add_format({'num_format': 'mm/dd/yyyy',
 'align': 'left'})

def to_dt(date):
 return dt.datetime.strptime(date, '%m/%d/%Y')

pd_csv = pd.read_csv('sample-data.csv')

print(pd_csv)

pd_csv['Date'] = pd_csv['Date'].apply(to_dt)

print(pd_csv[['Price', 'Piece']].groupby(pd_csv['Date'].dt.isocalendar().week).sum())

with open('sample-data.csv') as csv_file:
 csv_reader = csv.reader(csv_file, delimiter=',')

 # pd_df = pd.DataFrame(ind)
 print(pd_csv)

 header = ('Date', 'Price', 'Pieces')

 # Header

 for idx, row in enumerate(csv_reader):
 if idx == 0:
 for ridx, each in enumerate(row):
 worksheet.write(idx, ridx, each)
 else:
 for ridx, col in enumerate(row):
 if ridx == 0:
 data = dt.datetime.strptime(col, '%m/%d/%Y')
 worksheet.write(idx, ridx, data, date_format)
 else:
 worksheet.write(idx, ridx, int(col))

workbook.close()

Prettytable for commandline.

from prettytable import PrettyTable
x = PrettyTable()

test_data = {
 'a1': {
 'url': 'https://sec.domain.com',
 'preset1': True,
 'preset2': True,
 'preset3': False,
 'preset4': True,
 'preset5': True,
 'preset6': False,
 'preset7': True,
 },
 'a2': {
 'url': 'https://sec.domain.com',
 'preset1': True,
 'preset2': True,
 'preset3': False,
 'preset4': True,
 'preset5': True,
 'preset6': False,
 'preset7': True,
 },
 'a3': {
 'url': 'https://sec.domain.com',
 'preset1': True,
 'preset2': True,
 'preset3': False,
 'preset4': True,
 'preset5': True,
 'preset6': False,
 'preset7': True,
 },
 'a4': {
 'url': 'https://sec.domain.com',
 'preset1': True,
 'preset2': True,
 'preset3': False,
 'preset4': True,
 'preset5': True,
 'preset6': False,
 'preset7': True,
 },
}

x.field_names = test_data['a1'].keys()
for col, cval in test_data.items():
 x.add_row(cval.values())
print(x)

Mariadb/sqlalchemy to Pandas

from sqlalchemy import create_engine, text
import pandas as pd

engine = create_engine(
 "mariadb+pymsql://user:pass@some_mariadb/dbname?charset=utf8mb4", echo=True, future=True)

with engine.connect() as conn:
 sql_query = text("select * from table")
 df = pd.read_sql(sql_query, conn, index_col="id")

############################

import pandas as pd
import mariadb

mysql_connection = mariadb.connect(host='localhost',
 user='root',
 password='',
 database='tutorfall2016',)

sql = "SELECT * FROM sign_ins"
pd.read_sql(sql, mysql_connection, index_col='id')

Selenium Python Notes

Fri, 08 Jan 2021 00:00:00 +0000

Joting note that might be useful.

Selenium Interaction with Svelte Link and Button

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait

def wait_until_element_found(xpath):
 WebDriverWait(browser, timeout=10).until(
 lambda x: browser.find_element_by_xpath(xpath)
 )

 # To handle scrolling just incase
 last_coordinate = 300
 while not browser.find_element_by_xpath(xpath).is_displayed():
 print(last_coordinate)
 browser.execute_script(
 "window.scrollTo(0, {0})".format(last_coordinate))
 last_coordinate += 300
 sleep(1)
 sleep(1)

 # For debugging and tracking
 element = browser.find_element_by_xpath(xpath)
 print(element.location)

 browser.find_element_by_xpath(xpath).click()

browser = webdriver.Firefox()
browser.maximize_window()
browser.get('https://tech.moe.ph/')
wait_until_element_found("//button[contains(@class,'cursor-pointer')]")
wait_until_element_found("//a[.='Tags']")
wait_until_element_found("//li[.='live-system']")
wait_until_element_found(
 "//h1[starts-with(., 'Using Persistent Live System as Poor Man')]")

# xpath with contains
# wait_until_element_found(
# "//h1[contains(text(), 'Persistent Live System as Poor']")
# wait_until_element_found(
# "//*[contains(text(), 'Persistent Live System as Poor']")
# Reference
# https://stackoverflow.com/questions/12323403/how-do-i-find-an-element-that-contains-specific-text-in-selenium-webdriver-pyth
# driver.find_elements_by_xpath("//*[contains(text(), 'My Button')]")

wait_until_element_found("//a[.='debian']")
wait_until_element_found(
 "//h1[starts-with(., 'Using Persistent Live System as Poor Man')]")
wait_until_element_found(
 "//h1[starts-with(., 'Using Persistent Live System as Poor Man')]")
wait_until_element_found(
 "//button[.='Back']")
browser.quit()
print("Done!")

Selenium with Form Interaction

from selenium import webdriver
# from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait, Select
from time import sleep

browser = webdriver.Firefox()
#browser = webdriver.Chrome()
browser.maximize_window()
# browser.get(
# 'file:///home/tonydeguia/Downloads/templated-industrious/elements.html')
browser.get(
 'https://templated.co/items/demos/industrious/elements.html')

def wait_until_element_found(xpath):
 # browser.execute_script("window.scrollTo(0, 0)")
 WebDriverWait(browser, timeout=10).until(
 lambda x: browser.find_element_by_xpath(xpath)
 )

 # To handle scrolling just incase
 # last_coordinate = 300
 # while not browser.find_element_by_xpath(xpath).is_displayed():
 # print(last_coordinate)
 # browser.execute_script(
 # "window.scrollTo(0, {0})".format(last_coordinate))
 # last_coordinate += 300
 # sleep(1)
 sleep(1)

 # For debugging and tracking
 element = browser.find_element_by_xpath(xpath)
 print(element.location)
 # browser.execute_script("window.scrollTo({0}, {1})"
 # .format(element.location['x'] + 20, element.location['y'] + 20))

 browser.find_element_by_xpath(xpath).click()

def wait_until_then_select(xpath, choice):
 # browser.execute_script("window.scrollTo(0, 0)")
 WebDriverWait(browser, timeout=10).until(
 lambda x: browser.find_element_by_xpath(xpath)
 )
 sleep(1)

 Select(browser.find_element_by_xpath(xpath)).select_by_value(choice)

def wait_until_then_fill(xpath, data):
 WebDriverWait(browser, timeout=10).until(
 lambda x: browser.find_element_by_xpath(xpath)
 )
 sleep(1)

 element = browser.find_element_by_xpath(xpath)
 print(element.location)

 browser.find_element_by_xpath(xpath).send_keys(data)

def wait_until_then_submit(xpath):
 WebDriverWait(browser, timeout=10).until(
 lambda x: browser.find_element_by_xpath(xpath)
 )
 sleep(1)

 element = browser.find_element_by_xpath(xpath)
 print(element.location)

 browser.find_element_by_xpath(xpath).submit()

# select by value
wait_until_then_select("//select[@id='category']", "epsilon")

# Checkbox button
# Use element instead of id for checkbox
wait_until_element_found("//label[.='Checkbox alpha']")
wait_until_element_found("//label[.='Checkbox beta']")

# Check if True
# driver.element().is_selected()

# Radio select
# Use element instead of id for radio select
wait_until_element_found("//label[.='Radio gamma']")

# Text field select
wait_until_then_fill("//input[@name='name']", "superusername")
wait_until_then_fill("//input[@name='email']", "superemailaddress")

# Submit
wait_until_then_submit("//input[@value='Submit Form']")

sleep(10)
browser.quit()
print("Done!")

browser.get('https://www.w3schools.com/bootstrap/bootstrap_modal.asp')
wait_until_element_found("//button[.='Click To Open Modal']")
browser.switch_to.active_element
wait_until_element_found("//p[.='Some text in the modal.']")
wait_until_element_found("//button[.='Close']")

Extended Selenium Class

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait, Select
from time import sleep
import sys

class WebNavigator(webdriver.Firefox, webdriver.Chrome):
 sleep_int = 0.5
 webdriver_timeout = 10

 def __init__(self, browser):
 if browser == "Firefox":
 webdriver.Firefox.__init__(self)
 elif browser == "Chrome":
 webdriver.Chrome.__init__(self)
 else:
 sys.exit("Please specify webdriver (Firefox or Chrome).")
 self.maximize_window()

 def wait_then_click(self, xpath):
 WebDriverWait(self, timeout=self.webdriver_timeout).until(
 lambda x: self.find_element_by_xpath(xpath)
 )
 self.find_element_by_xpath(xpath).click()
 sleep(self.sleep_int)

 def wait_then_select(self, xpath, choice):
 WebDriverWait(self, timeout=self.webdriver_timeout).until(
 lambda x: self.find_element_by_xpath(xpath)
 )
 Select(self.find_element_by_xpath(xpath)).select_by_value(choice)
 sleep(self.sleep_int)

 def wait_then_fill(self, xpath, data):
 WebDriverWait(self, timeout=self.webdriver_timeout).until(
 lambda x: self.find_element_by_xpath(xpath)
 )
 self.find_element_by_xpath(xpath).send_keys(data)
 sleep(self.sleep_int)

Assorted helper for Selenium

import argparse

parser = argparse.ArgumentParser(description="""
 Parser
 """)
parser.add_argument("username", help="Account login")
parser.add_argument("password", help="Account password")
parser.add_argument("-b", "--browser",
 help="Webdriver to be used. Default: Chrome")
group = parser.add_mutually_exclusive_group()
group.add_argument(
 "-a", "--all", help="Execute to all regions", action="store_true")
group.add_argument("-r", "--region", help="Execute to this specific region")
args = parser.parse_args()
print(args.username)
print(args.password)
print(args.browser)

Selenium CSS Selector

[HTML TAG].[CLASS] [HTML TAG] [HTML TAG][ATTRIBUTE]

wait_until_css("div.row form label[for='checkbox-alpha']")
wait_until_css("div.row form label[for='checkbox-beta']")
wait_until_css_select("div.row form #category", "epsilon")

Single Sapper Front-End for Multiple Sites

Tue, 15 Dec 2020 00:00:00 +0000

If you have a Django-backend that running multiple sites, you might be thinking of a way to implement a single front-end for those sites. In my case, I am running multiple blog sites with different domains. This blog site is part of that single front-end design with Sapper.

To create this kind of setup, I decided to put all my api endpoints in /api/v1 of my domains, to prevent complications in using CloudFlare and CORS.

Cloudflare - they only allow 1 depth subdomain for their CDN service.
CORS - if you haven’t setup the CORS header, browsers would not render your api responses.

For this setup, I use an environment variable for easier configuration.

export API_ENDPOINT=/api/v1
export API_PROTOCOL=https

src/server.js

const { PORT, NODE_ENV, API_ENDPOINT, API_PROTOCOL } = process.env;

I declare API_ENDPOINT to indicate specific url, usually /api/v1, and for API_PROTOCOL, I just use this to indicate if my API_ENDPOINT is using http or https.

There is some logic in determining the apiEndpoint, since in development, I usually use a separate host for back-end.

To add this to middleware session, set this up into your sapper.middleware.

 sapper.middleware({
 session: (req, res) => {
 let apiEndpoint
 if (API_ENDPOINT.startsWith('http')) {
 apiEndpoint = API_ENDPOINT
 }
 else {
 apiEndpoint = [API_PROTOCOL, '://', req.headers.host, API_ENDPOINT].join('')
 }
 return ({
 apiEndpoint: apiEndpoint,
 })
 }
 })

Once this is setup, the apiEndpoint variable would be available during the preload part of web loading.

This is the code snipplet of one of my Svelte page.

<script context="module">
 export async function preload({ params }, session) {
 return this.fetch(`${session.apiEndpoint}/`)
 .then(r => r.json())
 .then(response => {
 return { posts: response };
 });
 }
</script>

Using the session.apiEndpoint during fetching instead of hardcording your API Endpoint in your every Sapper deployment would make it available to multiple domains. It is because in production build, it would use the request.headers.host variable. So if someone request the blog.moe.ph page, it would resolve into endpoint.

With this kind of setup, you can use a single front-end for multiple identical back-end application.

Building Live CD of Debian Testing Repository

Tue, 28 Jan 2020 00:00:00 +0000

Using testing release of Debian as live CD distribution is not as straight forward as building with stable release. The reason for that is there are some steps that should be taken first, before you can use the testing repository. Refer to this link for comprehensive guide for switching from stable to testing release.

These are the steps given in the documentation.

Edit your , changing stable (or buster, the current codename for stable) to testing (or bullseye, the current code name for the next stable release).

Remove, disable or comment out your stable security updates apt sources (anything with security.debian.org in it).

Remove, disable or comment out any other stable-specific apt sources, like *-backports or *-updates.

Verify that your installation is not fixed to a specific release in /etc/apt/apt.conf.d/00default-release.

In short, you must also integrate these requirements on your live-build configuration.

Step 1

Change the distribution related configuration in the config. To change the required settings, run the commands below. I used the testing release instead of bullseye so that I can still use the configuration once the bullseye release is considered as stable.

lb config -d testing;
lb config --debian-installer-distribution testing;
lb config --parent-distribution testing;
lb config --parent-debian-installer-distribution testing;

Step 2

Disable the security updates source.

lb config --security false;

Step 3

Disable the backport and updates sources.

lb config --backports false;
lb config --updates false;

Step 4

You also need to add the unstable repository on your archive sources, since some packages are not yet available on the testing release repository. You can do that by adding a configuration on config/archives/. Your archives folder should have the files below with corresponding settings. Take note that you should pin the unstable release to lower priority. Setting this incorrectly may cause some issues on the prioritized packages.

# cat config/archives/unstable.list.chroot
deb http://deb.debian.org/debian/ unstable main contrib non-free
# cat config/archives/unstable.pref.chroot
Package: *
Pin: release a=unstable
Pin-Priority: 100

Once you completed the give steps above, you should be able to successfully build a live CD from testing release.

Raspberry Pi 3 with Separate Power Supply on SIM800L

Sun, 05 Jan 2020 00:00:00 +0000

After initial testing with SIM800L without separate power supply, I explored a bit and bought additional components for my testing.

DC Buck Converter
12V 1A DC Adapter
DC Jack Barrel
Alligator Clips
Male to Female Jumper Wires
Quad-band Wired Cellular Antenna
Digital Multimeter

To set this all up, I assume that you already setup the GSM module with RPI3. We will just setup the power supply side of it.

Part 1

Connect the DC barrel jack with the 12V DC Adapter. Connect the DC barrel jack to DC buck converter using alligator clips. Then plug the DC adapter to outlet.

Check if the buck converter is powered on after plugging in. It has a LED indicator, so it should lit up.

We should setup the voltage output of the buck converter to 5V. To do this, you will see a screw on top of the blue part of the buck converter. Use multimeter to monitor it’s current voltage on the output, then turn the screw. Adjust it accordingly until it reads within ~5V area.

Part 2

Plug out the power supply for now. Get some male to female jumper wires. Connect the female jumpers to 5VIN and GND. After that, connect the male jumper of 5VIN with the out+ of the buck converter. Connect the male jumper of GND to out-. For solderless setup, I just use an extra alligator just to keep the male jumper in place.

Once it’s done, power-on your RPI3 first, then plug in the power of the GSM module. The GSM module should blink now, indicating that it’s already power on.

Note

If you noticed that the GSM module is suddenly turning off, then turn on, you might need to adjust the output of the buck converter. It usually happen when there’s not enough power, or too much voltage to the GSM module.

Part 3

Test the connection to the GSM module now. You should be able to connect to it seamlessly now.

Part 4 (Bonus)

Since we have more power to our GSM, we should use that power to get more signal. To have more signal, use a longer cellular antenna. Just make sure that it has SMA connector.

To connect it, just remove the default antenna provided with GSM module, then put the new antenna.

SIM800L Integration with Raspberry Pi 3

Tue, 17 Dec 2019 00:00:00 +0000

In this past few days, I bought some IOT modules since I have a lying RPi3 around. I bought this RPi3 around 2016 to make a media box. Since I’m not using it as much as of now, I decided to tinker with it. One of the module that caught my attention was SIM800L module. It’s only cheap, so I guess why not?

So here’s my TL;DR in setting it up.

Download and install the latest copy of Rasbpian to microSD.
Setup RPi to be able to use serial pins.
Connect the SIM800L to RPi GPIOs.
Test the serial connection to the module.

Here’s the needed peripherals to be able to finish the setup.

RPi3
MicroSD (16GB)
SIM800L Module
5 Female-to-Female pins
SIM Card

Part 1

To start, go and download the latest image of Rasbian from Raspberry official site. Just download the Raspian Buster Lite version, or go with the desktop one if you’re planning to connect it to TV or monitor.

After downloading, extract it. Then flash it to your flash disk. You can just use “dd” image utility, since it’s pretty easy. Please take note that everything will be overwritten on the microSD card.

dd if=[RASPBIAN IMAGE] of=/dev/[MICROSD CARD] bs=4M

After flashing it, go and put it to the RPi. It would be much better if you can connect the RPi to monitor or TV for some initial setup. Upon checking, it seems that SSHD service is not enabled by default. You would need to enable it for easier access to the Pi. It would also be best if you can connect it to your network via ethernet port.

Part 2:

Go boot your RPi. Once the server has been successfully booted, login into the server using this credentials.

Username: pi
Password: raspberry

After that, go and set-up these things below.

Enable SSHD service.
Upgrade the system to the latest version
Disable serial console of the RPi.
Enable UART.

Run the command below to enable sshd and do system upgrade if needed.

sudo systemctl enable sshd && sudo systemctl start sshd
sudo apt-get update && sudo apt-get upgrade -y

To disable the serial console, go run the command below. Then go to Option 5 - Interfacing Options. Go to P6 - Serial. Disable both the login shell, and the serial port hardware.

raspi-config

To enable UART, add the ’enable_uart=1’ parameter to ‘/boot/config.txt&’ file.

After setting everything up, go and power-off the RPi.

Part 3

Get your 5 pins. There are 5 pins that should be connected from the SIM800L module to RPi. Just make sure that RPi is off before connecting the pins. Be better sure than having a fried RPi board. Connect the SIM card as well before putting the pins.

5V Input (SIM) > 5V Pin (RPi)
Power Ground (SIM) > Ground (RPi)
RX Pin (SIM) > TX Serial Pin - GPIO 14 (RPi)
TX Pin (SIM) > RX Serial Pin - GPIO 15 (RPi)
Data Ground (SIM) > Ground (RPi)
I’m not really sure about the RST Pin on the SIM800L, but it seems that it’s not needed yet.

Note

There’s a lot of recommendation to have a separate power supply for SIM800L module, but since I don’t have those tools yet, I dared to connect it directly to RPi GPIOs. I didn’t have any issues yet as of writing this post. Separate power supply might be needed if you’re planning to connect to Internet while the RPi is up.

Note

Actually tested using the GSM module with Gammu and PPP, and it didn’t work as much. After testing with separate power supply, the module work as expected. It does need separate power supply.

Once everything is connected, go and power it on. Once the RPi is on, the module’s light should be on. There’s a separate blinking light indicator for the network connection.

Part 4

You need to confirm now if the SIM800L is responding to serial commands. To test it, I just use a pyserial module from Python. You can also use minicom tool, but I think I’m having issues using it.

You can download the needed module using the command below.

sudo apt-get install -y python3-pip && sudo pip3 install pyserial

Once downloaded, you can use this code to test the serial connection. Save it as ‘at-command.py’.

#This is the pyserial module
import serial

# Enable Serial Communication
# You may need to change the ttyS0 device and baudrate, but it worked in my case.
port = serial.Serial("/dev/ttyS0", baudrate=115200, timeout=1)

# Transmitting AT Commands to the Modem
# '\r\n' indicates the Enter key
port.write(b'AT\r\n')
rcv = port.read(10)
print(rcv)

Then run the script using the command below.

sudo python3 at-command.py

The command should print something like this.

b'AT\r\r\nOK\r\n'

Once you confirmed that the serial command is printing OK result, you can now successfully connect to the SIM800L module!

Congratulations, you can now use the SIM800L module to connect to Internet via ppp or send and receive sms using gammu. For these use cases, I’ll create another post for those.

Troubleshooting Notes

The at-command.py didn’t generate an OK result.

You might confused the RX and TX connections between the SIM800L module and RPi. You need to connect the RX of module to TX of RPi, and vice versa. It’s easy to encounter this issue, since I experienced it myself.

To Do

I’ll add setup photos later.
Add the references links.
Create another post for pppd and gammu setup.

Encrypted Persistent Filesystem in Live Debian

Mon, 09 Dec 2019 00:00:00 +0000

In the past few months, I really struggled to make my persistent storage encrypted. Every time I rebuilt my system and tested it, dm-crypt module missing error always pops-out during boot-up. Finally, I have able to make it work. Here’s how I fix it.

Initially, including these packages will just easily make the LUKS integration works.

cryptsetup
cryptsetup-run
cryptsetup-bin
cryptsetup-initramfs

Installing cryptsetup is indicated on the live-build manual on Debian site. I also include cryptsetup-initramfs, since it’s probably required. Having crypto module in initramfs is a must, since decrypting the volumes would happen in early stage of boot-up.

Unfortunately, once you built the image and test it, it would not work.

After testing for several months, I decided to check the documentation of cryptsetup-initramfs, and the hooks during chroot build. Upon checking, I noticed these two configuration files.

root@debian:~# find /etc -name *cryptsetup*
/etc/cryptsetup-initramfs
/etc/initramfs-tools/conf.d/cryptsetup

The /etc/initramfs-tools/conf.d/cryptsetup file was generated by the 0030 hook. It seems fine, but I noticed cryptsetup-initramfs. Upon checking, there is a separate configuration about the same settings as cryptsetup, but the CRYPTSETUP was not set.

root@debian:~# cat /etc/cryptsetup-initramfs/conf-hook
#
# Configuration file for the cryptroot initramfs hook.
#

#
# CRYPTSETUP: [ y | n ]
#
# Add cryptsetup and its dependencies to the initramfs image, regardless
# of _this_ machine configuration. By default, they're only added when
# a device is detected that needs to be unlocked at initramfs stage
# (such as root or resume devices or ones with explicit 'initramfs' flag
# in /etc/crypttab).
# Note: Honoring this setting will be deprecated in the future. Please
# uninstall the 'cryptsetup-initramfs' package if you don't want the
# cryptsetup initramfs integration.
#

# CRYPTSETUP=

Since I have nothing more to lose, I set the cryptsetup-initramfs settings by overwriting the configuration via includes.chroot. I have make sure that CRYPTSETUP=y is in the file.

I have proceed to rebuilding it again, and I noticed that the weird cryptsetup error during initramfs compilation process is gone. Seems good, right?

I download the new image, burn it to my flashdrive, setup persistence storage, then boot. Guess what? The cryptsetup prompt me to my LUKS password. The thing was able to work finally!

Findings

If you are planning to use encrypted persistent storage, always enable the CRYPTSETUP in cryptsetup-initramfs settings. It will make your life easier. You can enable it via hooks, or overwrite it with your custom includes.chroot file. For me, I just use the includes.chroot option to reduce clutter in my hooks.

Here’s the reference for this.

Insurance: You're Initial Step To Financial Stability

Sun, 17 Nov 2019 00:00:00 +0000

Stock market is always fascinating. When you think that you are playing with the market, it is always the other way around. Market usually play with you, and when that happens, your portfolio bleeds.

As for me, I am risk-taker, and plunge into the market. I always believe that experience is the best teacher. This is the case with the stock market, with a cost. Every move you do on market cost money, and if you’re not careful, you will dive into loss spree. It might be alright for someone who is risk-tolerant, but what about beginners? That’s where Investragrams Trading Cup come to play.

Investagrams Trading Cup is a trading competition in the Philippines. You can trade with virtual money, basing all price movements on the Philippine Stocks Exchange. This is a great place to learn, without actually risking your hard-earned money. Investragrams also provide learning modules. It will further help you learn how the stock market moves.

If you consider yourself entering the stock market in the future, or curious to learn new stuff like stock market and portfolio management, this trading cup will be the best place to learn and play.

To check the Trading Cup details, you can check the link below.

Bored Opinion: Sexfriend

Sat, 16 Nov 2019 00:00:00 +0000

The idea of sexfriend still facinates me. Having a sexfriend is cool, as long as you and your partner is only messing with 1 or 2 partners at a time. Having multiple partners are dangerous, to be honest.

This kind of relationship is convenient as well. You don’t need to invest a lot of time in your partner, and just meet with them when you just want to have some skinships or sex. Few things to invest, few things to lose.

I guess there are a lot of people that don’t really want to commit with someone, but still want to have a fulfilling sex life.

For me, I don’t really get it when someone enter a relationship without marriage in sight. Isn’t it just a waste of time? Waste of effort? Man, if I want to settle, I’ll just enter a relationship and make it clear to my partner that I’m already considering her as a marriage partner. If you don’t see me on the same light, then I guess let’s just settle being a sexfriend or friends, and don’t waste both of our time.

Just don’t be a whore or a manwhore when entering a sexfriend relationship. Having 2 partners, then changing partners every month seems like a different thing than that sexfriend thing.

Last thing, if you are entering this kind of relationship, just make sure to make some rules and stick to it. Someone break a rule? Break the relationship. It’s pretty convenient setup, but it has also some risk. And don’t put some romantic feelings on it as well. You’re putting yourself to a lot of headache.

Using Persistent Live System as Poor Man's System

Sat, 16 Nov 2019 00:00:00 +0000

I have a weird setup of my work station. I am currently using a system with damaged SATA boards, so I search for a way to use a diskless work station. It’s not entirely diskless, since I’m now using a flashdrive with live Debian.

If you are interested, here is the link for the base configuration of my live system.

Building

The image will be built by using container. Use the Dockerfile to generate a container for live build. Buildah is used in creating the container image.

buildah bud -f Dockerfile -t debian-live .

Run the container using podman. The container needs privilaged access to the host system. You also need to use persistent storage for the image file, with exec,dev flags.

podman run -ti --privileged --name debian-live -v vdebian:/app:exec,dev debian-live

Go to /app directory. Clone this repository, and go inside the repository’s directory.

cd /app;
git clone https://github.com/ZeroExistence/debian-live-workstation.git;
cd debian-live-workstation;

Run the live build command.

lb build

Site Redirection in Traefik using RedirectRegex Middleware

Wed, 06 Nov 2019 00:00:00 +0000

Implementing URL redirection in Traefik works like normal service with middleware processing. To setup redirection, setup the base router with middleware, and some dummy service.

http:
 routers:
 shop-secure:
 entryPoints:
 - "web-fe-secure"
 rule: "Host(`shop.moe.ph`)"
 middlewares:
 - "shopredirect"
 service: "shop-redirect"
 tls: {}

For middleware, you will need to use redirectRegex middleware processor. Just put the URL that you want to process, and it’s replacement.

http:
 middlewares:
 shopredirect:
 redirectRegex:
 regex: "^https://shop.moe.ph/(.*)"
 replacement: "https://shopee.ph/moe_ph"

Unfortunately, this router-middleware processing of URL redirection still needs service block. If you are redirecting your URL to another domain, just put a dummy service.

http:
 services:
 shop-redirect:
 loadBalancer:
 servers:
 - url: "http://localhost/"

To do more complex redirection, just check the Traefik’s documentation.

Investagrams Trading Cup: Playground for the Curious

Fri, 30 Aug 2019 00:00:00 +0000

Stock market is always fascinating. When you think that you are playing with the market, it is always the other way around. Market usually play with you, and when that happens, your portfolio bleeds.

If you consider yourself entering the stock market in the future, or curious to learn new stuff like stock market and portfolio management, this trading cup will be the best place to learn and play.

To check the Trading Cup details, you can check the link below.

King Traefik 2.0, Queen Django, and Jack passHostHeader

Fri, 30 Aug 2019 00:00:00 +0000

Deploying Django application means deploying it behind Nginx webserver most of the time. We are also using this setup. Deploying it this way is easy, since you need to set proxy_pass and proxy_set_header.

The only problem with this setup is when you only have 1 Nginx webserver for multiple Django backend. When you deploy new Django apps, you need to add new setting for it and restart the server. You might be able to reload it by sending SIGHUP to Nginx master process, but we are using container, so it’s hassle. You also need to reload it every time you edited something.

That’s when we look into alternative reverse proxy. We look for more cloud-native proxy, and that’s when we found about Traefik. Traefik is a cloud native edge router. It has integrations into multiple cluster technology. It also supports live reload of configuration files for dynamic settings. We are using podman to manage containers, so we are utilizing the dynamic file reload.

We start playing with Traefik, and use version 2. Tinkering with Traefik is not that easy if you’re not familiar with the earlier version.

The setting that really ticked our testing is passing the proxy host header to backend. The first settings that we look through are middleware headers settings of Traefik. This is the most logical place to look first, if you are playing with headers, right? We played with these settings.

Seems okay, but it didn’t actually work. The host header is still not pass to the backend. I hold some grudge to Traefik this time, then I realized that I need to check the settings for earlier version. There are Traefik settings scattered through Github, so I found this boolean setting.

passHostHeader = true

I guess this setting should be present to Traefik 2.0. After checking the dynamic configuration reference, I found this setting.

services:
 Service01:
 loadBalancer:
 passHostHeader: true

I set this to true and the backend is now responding.

I guess everyone should know about this setting.

Bored Opinion: Suicide is a Good Thing

Tue, 20 Aug 2019 00:00:00 +0000

Suicide is such a cool thing. Isn’t it the last biggest decision that you can make when you’re still alive. Why would other prevents suicide, if the one who decided it is themselves?

I also like to point out that suicide is pretty much like abortion? Isn’t abortion worse than suicide? Atleast in suicide, you consent to die. You decided it. Do you consent when you are still a fetus, and your parents decided to abort you? Well, you didn’t consent to be born as well, so that’s kind of funny.

I also like the idea that you decide on suicide on selfish manner. You can only decide on it when you removed the other external factors like family and friends. I don’t care why you commit suicide, but it’s cool.

What if someone asked me for advice that they want to end it all? I’ll probably say “why not?”. If someone I know commit suicide, I’ll say “man, that’s cool and dope”.

Personally, suicide might be the end of me. Or maybe not. It’s too boring to kill myself, and I prefer to play the game of life a little bit longer. You are already suffering enough, so you should learn how to enjoy suffering.

Podman, Containerization, and a Poor Guy

Tue, 20 Aug 2019 00:00:00 +0000

Deploying Kubernetes, Docker, and cloud-native apps needs large type of servers, but what if you are a poor guy like me and just want to use containerization technology on small scale servers? One thing that can help you is Podman.

. Ahh, what a great daemonless container engine. Daemonless? Yes. It means that it has minimal footprint on the server, giving much power to those containers.

We currently use Podman to manage our apps on a 4GB/2vCPU Fedora 30 Linode. As of now, we are comfortably running these apps:

Reverse-proxy Nginx
Multiple Django sites
Tor relay node
Ipfs node
Syncthing node
Development containers

You might wonder, but running those multiple containers still consume a lot of resources. This is right, but one other great thing about container is resource limiting. I just simply limit the resources on all production containers to 2GB/1vCPU. I did this by running my containers as a service, then set-up an environment file with this configuration.

[root@prod ~]# grep -i EnvironmentFile /etc/systemd/system/cnginx.service
EnvironmentFile=-/etc/sysconfig/podman
[root@prod ~]# cat /etc/sysconfig/podman
PODMAN_OPTIONS=\"--rm -a stdout -a stderr -m 2g --cpus 1\"

This is to ensure that no one would hog everything, rendering others non-responsive. I also set them as a service to automate container restart if something happens.

[root@prod ~]# grep -i restart /etc/systemd/system/cnginx.service
Restart=always
RestartSec=15s",

There are instances that podman would kill container due to OOM, but restore the service by automated restart. If there’s multiple case of OOM, you will need to adjust the resources accordingly.

After using podman for several months now, I just love it. I wish that they would be able to push their binaries to Debian repository.

First Post on Our Blogs

Mon, 19 Aug 2019 00:00:00 +0000

Hello everyone, this is moe.ph and I would like to welcome you to our simple blog sites. I’m already planning to launch some kind of blog site in the past to help me practice my writing skills and to share some knowledge that I learned in everyday life.

I have launch three sites for our blogs, the blog.moe.ph, the tech.moe.ph, and the finance.moe.ph. I launched different blog sites, so that I can separate each of our posts to appropriate sites. I don’t want to post some shitposting with some technical posts.

For blog.moe.ph, I might tend to post some of my thoughts here and maybe some business and Facebook related stuff.

For tech.moe.ph, I would compile most of my computing related stuffs there like on how I setup this blog sites and my environment on running my systems

On finance.moe.ph, I would probably post most stocks-related stuff there and some pointers in making money. I’m still not sure if finance would be the best term for that blog site, but I guess whatever works!

Since I’m on a freelance mode right now, I might be able to post frequently on these blogs, so I hope that you would pick something out of these blogs.

I guess that’s all for now, and keep being cute and depressed.

Blogs | blog.moe.ph

XLSXWriter Pandas CSV Python Notes

Prettytable for commandline.

Mariadb/sqlalchemy to Pandas

Selenium Python Notes

Selenium Interaction with Svelte Link and Button

Selenium with Form Interaction

Selenium Modal Interaction

Extended Selenium Class

Assorted helper for Selenium

Selenium CSS Selector

Single Sapper Front-End for Multiple Sites

Building Live CD of Debian Testing Repository

Step 1

Step 2

Step 3

Step 4

Raspberry Pi 3 with Separate Power Supply on SIM800L

Part 1

Part 2

Part 3

Part 4 (Bonus)

SIM800L Integration with Raspberry Pi 3

Part 1

Part 2:

Part 3

Part 4

Troubleshooting Notes

To Do

Encrypted Persistent Filesystem in Live Debian

Findings

Insurance: You're Initial Step To Financial Stability

Bored Opinion: Sexfriend

Using Persistent Live System as Poor Man's System

Building

Site Redirection in Traefik using RedirectRegex Middleware

Investagrams Trading Cup: Playground for the Curious

King Traefik 2.0, Queen Django, and Jack passHostHeader

Bored Opinion: Suicide is a Good Thing

Podman, Containerization, and a Poor Guy

First Post on Our Blogs