tmp: 2018

Monday, November 26, 2018

UnicodeDecodeError on python3.5

Wednesday, November 21, 2018

nginx memo

### package

基本は `nginx` 、lua を使いたい場合は `nginx-extras` 
* http://blog.it.churaumi.tv/nginx-light-full-extra-configure-compare

### config file path
設定ファイルを置く場所について

サイト設定 `server` は `conf.d` に入れてもいいけど  
区別のために `sites-available` に置いて `sites-enabled` に symlink 貼った方が良さげ

lua ファイルは特に決まってないっぽいので `/etc/nginx/lua/` とかで

あとは基本的に↓
* https://gakumon.tech/nginx/nginx_conf_directory.html

### content cache

#### multi server
1台でキャッシュを扱う場合は特に何も考えなくていい

redis, memcached 等のストレージを利用するケースだとモジュールが用意されているがキャッシュの設定はアプリケーションで行わないといけない様なのでちょっと微妙
```
A response should be put in memcached in advance by means external to nginx.
```
* http://nginx.org/en/docs/http/ngx_http_memcached_module.html

```
The backend should set the data in redis. The redis key is /uri?args.
```
* https://www.nginx.com/resources/wiki/modules/redis/
* https://www.nginx.com/resources/wiki/modules/redis2/

ローカルキャッシュを利用する場合は下記の様な構成にするとうまくいく (`sharded cache` or `shared cache`)
* https://www.nginx.com/blog/shared-caches-nginx-plus-cache-clusters-part-1/

#### purge cache
nginx plus(有償) だと proxy_cache_purge module が提供されている

無料版だと
* [Cache Purge](https://github.com/FRiCKLE/ngx_cache_purge)
* [Selective Cache Purge](https://github.com/wandenberg/nginx-selective-cache-purge-module)

ここら辺が使えそう（ただしnginxをソースからビルドしないといけないっぽいのでメンテとか考えると面倒そう）

または lua で自力でやる場合は
* https://scene-si.org/2016/11/02/purging-cached-items-from-nginx-with-lua/
* https://scene-si.org/2017/01/08/improving-nginx-lua-cache-purge/

ただし `proxy_cache_key` を指定しても key が揺らぐ(Vary, set-cookie とかで)みたいなのでそこら辺気にしないと確実に消せない  
デバイス別にキャッシュした場合も考慮する必要あり
```
        proxy_ignore_headers Cache-Control Expires Set-Cookie Vary;
        proxy_hide_header Set-Cookie;
```
example
* https://gist.github.com/fiahfy/a9d5ccbbe716286fe1b1ec71a65f1489

### proxy_set_header
location でheaderを追加したい時とかにそのまま追加だとうまくいかないのでいちいち全て再設定する必要がある  
`bad`
```
server {
:
    proxy_set_header Host $host;
:
    location / {
        proxy_set_header X-Foo 'foo';
    }
```
`good`
```
server {
:
    proxy_set_header Host $host;
:
    location / {
        proxy_set_header Host $host;
        proxy_set_header X-Foo 'foo';
    }
```

### template
#### log
```
log_format ltsv 'host:$http_host\t'
                'port:$server_port\t'
                'time:$time_iso8601\t'
                'remote_addr:$remote_addr\t'
                'request_method:$request_method\t'
                'request_length:$request_length\t'
                'request_uri:$request_uri\t'
                'https:$https\t'
                'uri:$uri\t'
                'query_string:$query_string\t'
                'status:$status\t'
                'bytes_sent:$bytes_sent\t'
                'body_bytes_sent:$body_bytes_sent\t'
                'referer:$http_referer\t'
                'useragent:$http_user_agent\t'
                'forwardedfor:$http_x_forwarded_for\t'
                'request_time:$request_time\t'
                'upstream_response_time:$upstream_response_time';
access_log /var/log/nginx/access.log ltsv;
error_log /var/log/nginx/error.log;
```

#### proxy cache
```
proxy_cache_path /var/cache/nginx/app
                 levels=1:2
                 keys_zone=app:8m
                 inactive=10m
                 max_size=128m;
proxy_temp_path  /var/cache/nginx/tmp;
```

#### map
```
map $http_user_agent $mobile {
    default     0;
    ~iPhone     1;
    ~Android    1;
    ~BlackBerry 1;
}
```

#### ssl
```
ssl_protocols TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_certificate /etc/ssl/certs/site.pem;
ssl_certificate_key /etc/ssl/private/site.key;
```

Thursday, November 1, 2018

fs performance (sync vs async)

```js
const fs = require('fs')
const path = require('path')
const util = require('util')

const calcSync = (filepath) => {
  const stats = fs.lstatSync(filepath)
  if (!stats.isDirectory()) {
    return [1, stats.size]
  }
  let totalNum = 0
  let totalSize = 0
  const filenames = fs.readdirSync(filepath)
  for (let filename of filenames) {
    const [num, size] = calcSync(path.join(filepath, filename))
    totalNum += num
    totalSize += size
  }
  return [totalNum, totalSize]
}

const calc = async (filepath) => {
  const stats = await util.promisify(fs.lstat)(filepath)
  if (!stats.isDirectory()) {
    return [1, stats.size]
  }
  let totalNum = 0
  let totalSize = 0
  const filenames = await util.promisify(fs.readdir)(filepath)
  for (let filename of filenames) {
    const [num, size] = await calc(path.join(filepath, filename))
    totalNum += num
    totalSize += size
  }
  return [totalNum, totalSize]
}

const src = '<target>'
;(async () => {
  console.time('calcSync')
  console.log(calcSync(src))
  console.timeEnd('calcSync')

console.time('calc')
  const r = await calc(src)
  console.log(r)
  console.timeEnd('calc')
})()
```

```
[ 23723, 359622209 ]
calcSync: 352.919ms
[ 23723, 359622209 ]
calc: 1406.648ms
```

Wednesday, September 19, 2018

Check TLS

### web
https://www.ssllabs.com/ssltest/

### curl
```bash
$ curl -v -s --tlsv1.0 "https://www.google.com" > /dev/null
* Rebuilt URL to: https://www.google.com/
*   Trying 2404:6800:400a:808::2004...
* TCP_NODELAY set
* Connected to www.google.com (2404:6800:400a:808::2004) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.0 (OUT), TLS handshake, Client hello (1):
} [220 bytes data]
* TLSv1.0 (IN), TLS handshake, Server hello (2):
{ [96 bytes data]
* TLSv1.0 (IN), TLS handshake, Certificate (11):
{ [2104 bytes data]
:
```
* `--sslv2` SSL2.0
* `--sslv3` SSL3.0
* `--tlsv1.0` TLS1.0
* `--tlsv1.1` TLS1.1
* `--tlsv1.2` TLS1.2

### nmap
```bash
brew install nmap
```
　
```bash
$ nmap --script ssl-enum-ciphers -p 443 www.google.com
Starting Nmap 7.70 ( https://nmap.org ) at 2018-09-19 14:14 JST
Nmap scan report for www.google.com (172.217.161.196)
Host is up (0.0097s latency).
Other addresses for www.google.com (not scanned): 2404:6800:400a:80b::2004
rDNS record for 172.217.161.196: kix07s03-in-f4.1e100.net

## 参考
* https://qiita.com/greymd/items/68b0c40044a88171235a
* https://www.cloudibee.com/disabling-tls-1-0-on-nginx/

Thursday, August 16, 2018

Web Push

Web Push (Firebase Cloud Messaging)

Web Push に Firebase Cloud Messaging を利用した場合の実装

### Set gcm_sender_id
Firebase project を作成して `103953800507` (固定値) を指定  
ドキュメントではこの固定値を入力するように記載されているが省略しても動作はする（謎）
`manifest.json`
```
{
  :
  "gcm_sender_id": "103953800507"
}
```

### initialize firebase app
Firebase project の設定値を指定  
`index.html`
```
<script src="https://www.gstatic.com/firebasejs/5.3.1/firebase.js"></script>
<script>
  // Initialize Firebase
  // TODO: Replace with your project's customized code snippet
  var config = {
    apiKey: "<API_KEY>",
    authDomain: "<PROJECT_ID>.firebaseapp.com",
    databaseURL: "https://<DATABASE_NAME>.firebaseio.com",
    storageBucket: "<BUCKET>.appspot.com",
    messagingSenderId: "<SENDER_ID>",
  }
  firebase.initializeApp(config)
</script>
```

`app.js`
```
// Retrieve Firebase Messaging object.
const messaging = firebase.messaging()
```

### register service worker
ServiceWorker を `/firebase-messaging-sw.js` とする場合は firebase がよろしくやってくれるので省略可  
`app.js`
```
navigator.serviceWorker.register('sw.js')
  .then((registration) => {
    messaging.useServiceWorker(registration)
  })
```

### request permission and get token
`app.js`  
```
messaging.requestPermission()
  .then(() => {
    console.log('Notification permission granted.')

messaging.getToken()
      .then((currentToken) => {
        if (currentToken) {
          sendTokenToServer(currentToken)
        } else {
          // Show permission request.
          console.log('No Instance ID token available. Request permission to generate one.')
          // Show permission UI.
          sendTokenToServer(false)
        }
      })
      .catch((err) => {
        console.log('An error occurred while retrieving token. ', err)
        sendTokenToServer(false)
      })
  })
  .catch((err) => {
    console.log('Unable to get permission to notify.', err)
  })
```

### receive push
タブの状態（フォアグラウンド/バックグラウンド）に応じて処理する場所が違うのでそれぞれ実装

#### receive on foreground
`app.js`
```
messaging.onMessage((e) => {
  console.log('Message received. ', e)

const title = 'Title'
  const options = {
    body: e.data.text
  }

new Notification(title, options)
})
```

#### receive on background
`sw.js`
```
// Give the service worker access to Firebase Messaging.
// Note that you can only use Firebase Messaging here, other Firebase libraries
// are not available in the service worker.
importScripts('https://www.gstatic.com/firebasejs/5.3.1/firebase-app.js')
importScripts('https://www.gstatic.com/firebasejs/5.3.1/firebase-messaging.js')

// Initialize the Firebase app in the service worker by passing in the
// messagingSenderId.
firebase.initializeApp({
  'messagingSenderId': "<SENDER_ID>"
})

// Retrieve an instance of Firebase Messaging so that it can handle background
// messages.
const messaging = firebase.messaging()

messaging.setBackgroundMessageHandler((e) => {
  console.log('Message received. ', e)
  // Customize notification here
  const title = 'Title'
  const options = {
    body: e.data.text
  }

return self.registration.showNotification(title, options)
})
```

### send push
`firebase-admin` を利用、  
Firebase の サーバアカウントキーをダウンロードし `key.json` にリネーム  
`'TOKEN'` に対象の登録トークンを指定  
`main.js`
```
const admin = require('firebase-admin')

const serviceAccount = require('./key.json')

admin.initializeApp({
  credential: admin.credential.cert(serviceAccount)
})

// See documentation on defining a message payload.
const message = {
  data: {
    text: 'Here is a payload!'
  },
  token: 'TOKEN'
}

// Send a message to the device corresponding to the provided
// registration token.
admin.messaging().send(message)
  .then((response) => {
    // Response is a message ID string.
    console.log('Successfully sent message:', response)
  })
  .catch((error) => {
    console.log('Error sending message:', error)
  })
```
送信
```
node main.js
```

Web Push (VAPID)

Web Push に VAPID を利用した場合の実装方法

### generate VAPID public/private keys
`generate_vapid_keys.js`
```
const webPush = require('web-push')

const vapidKeys = webPush.generateVAPIDKeys()

const keys = {
  publicKey: vapidKeys.publicKey,
  privateKey: vapidKeys.privateKey,
}

console.log(keys)
```
生成
```
node generate_vapid_keys.js
```

### register service worker
FCMの場合と変わらず  
`app.js`
```
navigator.serviceWorker.register('/sw.js')
  .then((registration) => {
    swRegistration = registration
  })
```

### subscribe
`applicationServerKey` に上記で生成した公開鍵を指定  
`app.js`
```
swRegistration.pushManager.subscribe({
  userVisibleOnly: true,
  applicationServerKey: "PUBLIC_KEY"
})
  .then((subscription) => {
    // send subscription to server
  })
```

### receive push
FCMの場合と変わらず  
`sw.js`
```
self.addEventListener('push', (e) => {
  console.log('[Service Worker] Push Received', e)

const title = 'Title'
  const options = {
    body: e.data ? e.data.text() : 'Empty payload'
  }

e.waitUntil(self.registration.showNotification(title, options))
})
```

### send push
暗号化周りが面倒なので `web-push` を利用  
`'PUBLIC_KEY'`, `'PRIVATE_KEY'` にそれぞれ生成した公開鍵/秘密鍵を指定  
`'SUBSCRIPTION'` は購読時に取得したものを指定  
`main.js`
```
const webPush = require('web-push')

const payload = 'Here is a payload!'

const options = {
  vapidDetails: {
    subject: 'mailto:example_email@example.com',
    publicKey: 'PUBLIC_KEY',
    privateKey: 'PRIVATE_KEY'
  },
  TTL: 60
}

webPush.sendNotification(
  'SUBSCRIPTION',
  payload,
  options
)
```
送信
```
node main.js
```

Web Push (GCM/FCM)

送信のみにFCMを利用した場合の実装方法  
Firefox のみの話でいうと FCM を利用しないためFCM周りの設定は省略できる  
(Chromeを考えると必須ではあるが)

### Set gcm_sender_id
Firebase project を作成して `送信者 ID` を指定  
`manifest.json`
```
{
  :
  "gcm_sender_id": "SENDER_ID"
}
```

### register service worker
`app.js`
```
navigator.serviceWorker.register('/sw.js')
  .then((registration) => {
    swRegistration = registration
  })
```

### subscribe
`app.js`
```
swRegistration.pushManager.subscribe({
  userVisibleOnly: true
})
  .then((subscription) => {
    // send subscription to server
  })
```
`userVisibleOnly` は現状必須だが Firefox だと省略できる

### receive push
`sw.js`
```
self.addEventListener('push', (e) => {
  console.log('[Service Worker] Push Received', e)

const title = 'Title'
  const options = {
    body: e.data ? e.data.text() : 'Empty payload'
  }

e.waitUntil(self.registration.showNotification(title, options))
})
```

### send push

#### without data payloads
ペイロードなしの通知は endpoint と サーバーキーの指定のみで送信できる  
サーバーキー は Firebase project に記載されているものを使用  
```
curl "END_POINT" --request POST --header "TTL: 60" --header "Content-Length: 0" --header "Authorization: key=SERVER_KEY"
```

#### with data payloads
ペイロードありの通知は暗号化周りが面倒なので `web-push` を利用  
`'SUBSCRIPTION'` は購読時に取得したものを指定  
`main.js`
```
const webPush = require('web-push')

const payload = 'Here is a payload!'

const options = {
  gcmAPIKey: 'SERVER_KEY',
  TTL: 60
};

webPush.sendNotification(
  'SUBSCRIPTION',
  payload,
  options
)
```
送信
```
node main.js
```

Tuesday, March 27, 2018

Fluentd memo

### 動作確認
設定ファイル編集  
`/etc/fluentd/fluentd.conf`
```
<source>
    type forward
</source>
<match debug.**>
    type stdout
</match>
```

Restart fluentd  
```
$ sudo service td-agent restart
```

飛ばしてみる
```
$ echo '{"json":"message"}' | fluent-cat debug.test
```
`fluent-cat` がない場合以下を確認
* `/usr/lib/fluent/ruby/bin/fluent-cat`
* `/opt/td-agent/embedded/bin/fluent-cat`

ログを見てみる
```
$ tail /var/log/fluentd/stdout.log
2018-03-27 10:17:02 +0900 debug.test: {"json":"message"}
```

### Output file
`/etc/fluentd/fluentd.conf`
```
<match debug.file.**>
    type file
    path /var/log/fluent/app
    time_slice_format %Y%m%d
    time_slice_wait 1m
    flush_interval 5s
    utc
</match>
```
再起動して確認
```
$ sudo service td-agent restart
$ echo '{"json":"message"}' | fluent-cat debug.file.test
$ tail /var/log/fluent/app.20180405_0.log
2018-04-05T02:42:38Z debug.file.test {"json":"message"}
```

* `time_format`  
fileに出力される時間のフォーマット  
default: ISO-8601
* `utc`
ファイル分割にUTCを使用するか  
default: localtime
* `compress`  
圧縮するかどうか  
default: no compression
* `time_slice_format`  
どのくらいの期間でファイルを分割するか  
default: `%Y%m%d%H`
* `time_slice_wait`  
ファイル分割時の待ち時間、どのくらい遅れを許容するか  
default: `10m`
* `flush_interval`
データをflushする間隔  
default: `60s`

## 参考
* http://hivecolor.com/id/37
* https://qiita.com/zaburo/items/dbd943d370afe8e4a304
* http://www.usupi.org/sysad/253.html
### output file
* https://docs.fluentd.org/v0.12/articles/out_file
* https://blog.tnmt.info/2012/10/19/about-fluentd-out-file-plugin/

tmp

Monday, November 26, 2018

UnicodeDecodeError on python3.5

Wednesday, November 21, 2018

nginx memo

Thursday, November 1, 2018

fs performance (sync vs async)

Wednesday, September 19, 2018

Check TLS

Thursday, August 16, 2018

Web Push

Web Push (Firebase Cloud Messaging)

Web Push (VAPID)

Web Push (GCM/FCM)

Tuesday, March 27, 2018

Fluentd memo

Monday, March 19, 2018

Android emulator の hosts 書き換え

Sunday, March 4, 2018

Windows10 on VirtualBox

Thursday, February 1, 2018

adb command

Tuesday, January 16, 2018

Provisioning profile "XXXXX" doesn't include signing certificate

Friday, January 12, 2018

Ansible cli