Thursday, February 14, 2019

Standard provisioning of Ubuntu 18.04 on cloud systems

Step-by-step guide

  1. Make sure your account has sudo access, or you have an SSH key in the root account.
  2. SSH into the VM / system & become sudo ( sudo su - or sudo su - root )
  3. If this isn't a system managed by Google or Azure, consider updating the .ssh/authorized_keys file with the other pertinent admin keys.
  4. Use nano or vim to edit /etc/sysctl.conf with the contents of Baseline /etc/sysctl.conf ; and modify as needed.
  5. Run the commands in Provisioning Sequence in-terminal.
  6. Use nano or vim to edit /etc/systemd/swap.conf to make any desired changes.
  7. systemctl enable systemd-swap && systemctl restart systemd-swap
  8. ucaresystem-core
  9. Check over the system, and reboot when ready.

Provisioning Sequence


sysctl -p && cd ~
wget https://github.com/Utappia/uCareSystem/releases/download/v4.4.0/ucaresystem-core_4.4.0_all.deb
apt update && apt -f -y install deborphan xterm haveged make git
systemctl restart haveged
dpkg -i ucaresystem-core_4.4.0_all.deb && rm ucaresystem-core_4.4.0_all.deb
cd /opt && git clone https://github.com/Nefelim4ag/systemd-swap.git && cd systemd-swap && make install && cd ~

Baseline /etc/sysctl.conf

######### Server sysctl
# Mix of wiki.mikejung.biz/Sysctl_tweaks + stuff from unquietwiki.blogspot.com
# Tested with Linux Kernel 4.16.x ; 6-6-2018
# Change 8192 to 1024 or 2048 on small systems
# Change 2048 to 512 or 1024 on small systems
# Change bbr to illinois (wired) or westwood (wireless / lossy), if on pre-4.10 kernel
fs.file-max = 4194304
net.core.default_qdisc=fq_codel
net.core.netdev_max_backlog=8192
net.core.rmem_max=16777216
net.core.somaxconn=2048
net.core.wmem_max=16777216
net.ipv4.tcp_base_mss=1024
net.ipv4.tcp_congestion_control=bbr
net.ipv4.tcp_max_syn_backlog=8192
net.ipv4.tcp_mtu_probing=1
net.ipv4.tcp_rmem=4096 12582912 16777216
net.ipv4.tcp_slow_start_after_idle=0
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_wmem=4096 12582912 16777216

Thursday, June 7, 2018

Two possible options for optimizing an ElasticSearch 5.x cluster

Been trying to help my boss manage a small Elasticsearch cluster. Performance has been a nagging issue on it; and a lot of optimization examples are old for our needs (running 5.6.x at the moment).

Less index refreshing

Elastic suggests changing the "refresh interval" to something other than its default one second index refresh on certain clusters. This forum post told me what to shove into Kibana / CURL: think I went with 15 or 20 seconds.
curl -XPUT localhost:9200/_settings -d '{
  "index": {
    "refresh_interval": "15s"
  }
}'

Setup Curator

Per this awesome Stack Overflow piece, you can wrap your head around how the indexing works. Some other stuff I came across suggests that you can shrink the older indices down to 1 segment, and gain performance in doing so. Turns out there's a tool to do this!
  1. With Python installed, use a command prompt to run pip install elasticsearch-curator
  2. Optional for Windows: you can go to your Python\Scripts directory, and copy the curator EXEs to a folder you want to save those and the config files in.
  3. Create the config files for your needs; put them in a subdirectory. There has to be at least two files: a curator.yml file, and a second file that has the actions. Past users of Ansible will know how to lay this out.
  4. Example command to use with a task manager / cron / etc: curator.exe --config config\curator.yml config\actions\actions.yml

config/curator.yml

---
# Remember, leave a key empty if there is no value.
client:
  hosts:
    - localhost
logging:
  loglevel: INFO
  logfile: 'path_to_logfile'
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

config/actions/actions.yml

---
# Remember, leave a key empty if there is no value.
actions:
  1:
    action: forcemerge
    description: >-
      Perform a forceMerge on selected indices to 'max_num_segments' per shard.
    options:
      max_num_segments: 1
      timeout_override:
      delay: 60
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 7
    - filtertype: forcemerged
      max_num_segments: 1
      exclude: True

Wednesday, June 6, 2018

Updated sysctl.conf for Kernel 4.x

# Mix of wiki.mikejung.biz/Sysctl_tweaks + stuff from unquietwiki.blogspot.com
# Tested with Linux Kernel 4.16.x ; 6-6-2018

# Change 8192 to 1024 or 2048 on small systems
# Change 2048 to 512 or 1024 on small systems
# Change bbr to illinois (wired) or westwood (wireless / lossy), if on pre-4.10 kernel

fs.file-max = 1048576
net.core.default_qdisc=fq_codel
net.core.netdev_max_backlog=8192
net.core.rmem_max=16777216
net.core.somaxconn=2048
net.core.wmem_max=16777216
net.ipv4.tcp_congestion_control=bbr
net.ipv4.tcp_max_syn_backlog=8192
net.ipv4.tcp_mtu_probing=1
net.ipv4.tcp_rmem=4096 12582912 16777216
net.ipv4.tcp_slow_start_after_idle=0
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_wmem=4096 12582912 16777216
vm.swappiness=1

Monday, May 7, 2018

Javascript Resources

I've been beating my skull the past two months trying to fix up a particular web application. In doing so, I've stumbled into yet another programming language crisis (see also Ruby in 2007-2008, or the Python 2->3 migration mess). At least this time around, the tools available are a lot more amenable to keeping newer stuff working on older platforms; albeit with some magic on the backend. Updated May 15, 2018 with more materials.

Monday, February 26, 2018

mcrypt for PHP 7

PHP 7.2 completely deprecated mcrypt. I found this out trying to get some third-party software to work. However, I did come across a solution, that worked for me at least.

1. Based on OS & PHP setup, install the following...

2. Use the following code in a PHP include / main PHP class for your app...

Formatted with CodeFormatter

 // Composer loading for PHPseclib  
 require_once 'autoload.php';  
 $loader = new \Composer\Autoload\ClassLoader();  
 $loader->addPsr4('phpseclib\\', __DIR__ . 'vendor/phpseclib/phpseclib/phpseclib');  
 $loader->register();  
   
 // Mcrypt-enablement  
 require_once 'vendor/mollie/polyfill-libsodium/bootstrap.php';  
 require_once 'vendor/phpseclib/mcrypt_compat/lib/mcrypt.php';  

Wednesday, February 14, 2018

Working Reverse Proxy in IIS

Too many references I see about IIS talk about using URL Rewrite to proxy requests to other applications. It's never worked right for me. However, I've found in the past day or so, references that break that impasse.

Procedure

  1. Install URL Rewrite & Application Request Routing (ARR) into your IIS installation.
  2. Create an empty directory where IIS can access it. This is where web.config will live.
  3. Create a virtual directory in IIS for your application. Name it & the path per the subdirectory of the website you're creating (ex: /webservice ). Use the empty directory you created earlier.
  4. Replace the contents of web.config of the directory "hosting" the virtual directory, with a modified version of the example here.

References

web.config

Change DOMAINSERVERNAME to your "external" URL. Change 3000 to whatever local port your other application uses.

Formatted with CodeFormatter


 <?xml version="1.0" encoding="UTF-8"?>  
 <configuration>  
   <system.webServer>  
     <rewrite>  
       <rules>  
         <rule name="ReverseProxyInboundRule1" stopProcessing="true">  
           <match url="(.*)" />  
           <action type="Rewrite" url="http://localhost:3000/{R:1}" />  
           <serverVariables>  
             <set name="HTTP_X_ORIGINAL_ACCEPT_ENCODING" value="{HTTP_ACCEPT_ENCODING}" />  
             <set name="HTTP_ACCEPT_ENCODING" value="" />  
           </serverVariables>  
         </rule>  
       </rules>  
       <outboundRules>  
         <rule name="ReverseProxyOutboundRule1" preCondition="ResponseIsHtml1">  
           <match filterByTags="A, Form, Img" pattern="^http(s)?://localhost:3000/(.*)" />  
           <action type="Rewrite" value="http{R:1}://DOMAINSERVERNAME/{R:2}" />  
         </rule>  
         <rule name="RestoreAcceptEncoding" preCondition="NeedsRestoringAcceptEncoding">  
           <match serverVariable="HTTP_ACCEPT_ENCODING" pattern="^(.*)" />  
           <action type="Rewrite" value="{HTTP_X_ORIGINAL_ACCEPT_ENCODING}" />  
         </rule>  
         <preConditions>  
           <preCondition name="ResponseIsHtml1">  
             <add input="{RESPONSE_CONTENT_TYPE}" pattern="^text/html" />  
           </preCondition>  
           <preCondition name="NeedsRestoringAcceptEncoding">  
             <add input="{HTTP_X_ORIGINAL_ACCEPT_ENCODING}" pattern=".+" />  
           </preCondition>  
         </preConditions>  
       </outboundRules>  
     </rewrite>  
   </system.webServer>  
 </configuration>  


Thursday, November 16, 2017

Making sure a network share mounts after booting in Linux

I ran into a situation, where my CIFS / Samba share wasn't mounting automatically on boot. It appears to be a race condition: one that's resolvable by forcing another attempt later in the boot process. Create /etc/systemd/system/mountall.service, and use systemctl enable mountall to activate it.

[Unit]
Description=Ensure all drives are mapped
After=network.target

[Service]
Type=simple
ExecStartPre=/bin/sleep 5
ExecStart=/bin/mount -a

[Install]
WantedBy=multi-user.target

References