Thursday, June 7, 2018

Two possible options for optimizing an ElasticSearch 5.x cluster

Been trying to help my boss manage a small Elasticsearch cluster. Performance has been a nagging issue on it; and a lot of optimization examples are old for our needs (running 5.6.x at the moment).

Less index refreshing

Elastic suggests changing the "refresh interval" to something other than its default one second index refresh on certain clusters. This forum post told me what to shove into Kibana / CURL: think I went with 15 or 20 seconds.
curl -XPUT localhost:9200/_settings -d '{
  "index": {
    "refresh_interval": "15s"
  }
}'

Setup Curator

Per this awesome Stack Overflow piece, you can wrap your head around how the indexing works. Some other stuff I came across suggests that you can shrink the older indices down to 1 segment, and gain performance in doing so. Turns out there's a tool to do this!
  1. With Python installed, use a command prompt to run pip install elasticsearch-curator
  2. Optional for Windows: you can go to your Python\Scripts directory, and copy the curator EXEs to a folder you want to save those and the config files in.
  3. Create the config files for your needs; put them in a subdirectory. There has to be at least two files: a curator.yml file, and a second file that has the actions. Past users of Ansible will know how to lay this out.
  4. Example command to use with a task manager / cron / etc: curator.exe --config config\curator.yml config\actions\actions.yml

config/curator.yml

---
# Remember, leave a key empty if there is no value.
client:
  hosts:
    - localhost
logging:
  loglevel: INFO
  logfile: 'path_to_logfile'
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

config/actions/actions.yml

---
# Remember, leave a key empty if there is no value.
actions:
  1:
    action: forcemerge
    description: >-
      Perform a forceMerge on selected indices to 'max_num_segments' per shard.
    options:
      max_num_segments: 1
      timeout_override:
      delay: 60
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 7
    - filtertype: forcemerged
      max_num_segments: 1
      exclude: True

Wednesday, June 6, 2018

Updated sysctl.conf for Kernel 4.x

# Mix of wiki.mikejung.biz/Sysctl_tweaks + stuff from unquietwiki.blogspot.com
# Tested with Linux Kernel 4.16.x ; 6-6-2018

# Change 8192 to 1024 or 2048 on small systems
# Change 2048 to 512 or 1024 on small systems
# Change bbr to illinois (wired) or westwood (wireless / lossy), if on pre-4.10 kernel

fs.file-max = 1048576
net.core.default_qdisc=fq_codel
net.core.netdev_max_backlog=8192
net.core.rmem_max=16777216
net.core.somaxconn=2048
net.core.wmem_max=16777216
net.ipv4.tcp_congestion_control=bbr
net.ipv4.tcp_max_syn_backlog=8192
net.ipv4.tcp_mtu_probing=1
net.ipv4.tcp_rmem=4096 12582912 16777216
net.ipv4.tcp_slow_start_after_idle=0
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_wmem=4096 12582912 16777216
vm.swappiness=1

Monday, May 7, 2018

Javascript Resources

I've been beating my skull the past two months trying to fix up a particular web application. In doing so, I've stumbled into yet another programming language crisis (see also Ruby in 2007-2008, or the Python 2->3 migration mess). At least this time around, the tools available are a lot more amenable to keeping newer stuff working on older platforms; albeit with some magic on the backend. Updated May 15, 2018 with more materials.

Monday, February 26, 2018

mcrypt for PHP 7

PHP 7.2 completely deprecated mcrypt. I found this out trying to get some third-party software to work. However, I did come across a solution, that worked for me at least.

1. Based on OS & PHP setup, install the following...

2. Use the following code in a PHP include / main PHP class for your app...

Formatted with CodeFormatter

 // Composer loading for PHPseclib  
 require_once 'autoload.php';  
 $loader = new \Composer\Autoload\ClassLoader();  
 $loader->addPsr4('phpseclib\\', __DIR__ . 'vendor/phpseclib/phpseclib/phpseclib');  
 $loader->register();  
   
 // Mcrypt-enablement  
 require_once 'vendor/mollie/polyfill-libsodium/bootstrap.php';  
 require_once 'vendor/phpseclib/mcrypt_compat/lib/mcrypt.php';  

Wednesday, February 14, 2018

Working Reverse Proxy in IIS

Too many references I see about IIS talk about using URL Rewrite to proxy requests to other applications. It's never worked right for me. However, I've found in the past day or so, references that break that impasse.

Procedure

  1. Install URL Rewrite & Application Request Routing (ARR) into your IIS installation.
  2. Create an empty directory where IIS can access it. This is where web.config will live.
  3. Create a virtual directory in IIS for your application. Name it & the path per the subdirectory of the website you're creating (ex: /webservice ). Use the empty directory you created earlier.
  4. Replace the contents of web.config of the directory "hosting" the virtual directory, with a modified version of the example here.

References

web.config

Change DOMAINSERVERNAME to your "external" URL. Change 3000 to whatever local port your other application uses.

Formatted with CodeFormatter


 <?xml version="1.0" encoding="UTF-8"?>  
 <configuration>  
   <system.webServer>  
     <rewrite>  
       <rules>  
         <rule name="ReverseProxyInboundRule1" stopProcessing="true">  
           <match url="(.*)" />  
           <action type="Rewrite" url="http://localhost:3000/{R:1}" />  
           <serverVariables>  
             <set name="HTTP_X_ORIGINAL_ACCEPT_ENCODING" value="{HTTP_ACCEPT_ENCODING}" />  
             <set name="HTTP_ACCEPT_ENCODING" value="" />  
           </serverVariables>  
         </rule>  
       </rules>  
       <outboundRules>  
         <rule name="ReverseProxyOutboundRule1" preCondition="ResponseIsHtml1">  
           <match filterByTags="A, Form, Img" pattern="^http(s)?://localhost:3000/(.*)" />  
           <action type="Rewrite" value="http{R:1}://DOMAINSERVERNAME/{R:2}" />  
         </rule>  
         <rule name="RestoreAcceptEncoding" preCondition="NeedsRestoringAcceptEncoding">  
           <match serverVariable="HTTP_ACCEPT_ENCODING" pattern="^(.*)" />  
           <action type="Rewrite" value="{HTTP_X_ORIGINAL_ACCEPT_ENCODING}" />  
         </rule>  
         <preConditions>  
           <preCondition name="ResponseIsHtml1">  
             <add input="{RESPONSE_CONTENT_TYPE}" pattern="^text/html" />  
           </preCondition>  
           <preCondition name="NeedsRestoringAcceptEncoding">  
             <add input="{HTTP_X_ORIGINAL_ACCEPT_ENCODING}" pattern=".+" />  
           </preCondition>  
         </preConditions>  
       </outboundRules>  
     </rewrite>  
   </system.webServer>  
 </configuration>  


Thursday, November 16, 2017

Making sure a network share mounts after booting in Linux

I ran into a situation, where my CIFS / Samba share wasn't mounting automatically on boot. It appears to be a race condition: one that's resolvable by forcing another attempt later in the boot process. Create /etc/systemd/system/mountall.service, and use systemctl enable mountall to activate it.

[Unit]
Description=Ensure all drives are mapped
After=network.target

[Service]
Type=simple
ExecStartPre=/bin/sleep 5
ExecStart=/bin/mount -a

[Install]
WantedBy=multi-user.target

References





Wednesday, October 11, 2017

Attempts to fix 802.11ac WiFi networking issues

My boss and I have been dealing with WiFi issues in regards to various home & work situations. I've been trying to assemble a good strategy to deal with them, and there appears to be some promising leads.

  • 802.11ac environments, you have 5ghz available, as well as 2.4ghz. However, it turns out that "Dynamic Frequency Selection" is eager to not interfere with radar installations. Per a chart of 802.11 5ghz channels, that's channels 100-149 to avoid using in a major urban area; so I am currently not using that + have DFS turned off. Use a good WiFi analyzer app to pick the least used channel outside that range.
  • If you're supporting older iPhones for people, it may not like 40mhz 2.4ghz setups; found multiple posts online about that. Keep your 2.4ghz to 20mhz; and also to N-only mode if you've managed to ditch all your old B & G devices.
  • In a residential setting, turn the power down. And if you know your neighbors, ask them to as well. 50-75% should be fine; especially in an apartment complex.
  • I set my Fragmentation Threshold to 1500, and RTS Threshold to 1501 for now. I need to study these more, but your standard Ethernet framing isn't more than 1500 anyway; and I've read that in congested areas, "less is more".
  • I also read that if your devices aren't going to be switching SSIDs a lot, to try raising the DTIM above the default of 1 (2 or 3 seem to be valid), and/or raise your Beacon Interval past 100 (currently set mine at 3000). This is supposed to keep your wireless devices from waking up too much, and draining power. However, too high of a the settings are said to negatively impact realtime chat + video apps.
Added: good, but dense, eBook on this stuff

Added#2: some more suggested tweaking on the Frag/RTS setting. I'm trying 1312 for my settings now, since I do have IPv6 on my connection (1280 bytes +34 byte MAC header).